I'm trying to write a google cloud function that automatically downloads data from a certain kaggle data set and upload it to a google cloud bucket. No matter how I try to include my kaggle api key in my google cloud function, the kaggle api can't find it!
I uploaded my kaggle api key to the secret manager and I wrote some code to write the key value into a file stored in /tmp, and set KAGGLE_CONFIG_DIR to /tmp. I granted the cloud function secret manager admin permissions and created added a runtime service account with permission to access the kaggle key as well just to make sure
I reference the secret via a mount path that I set here
First I tried just setting KAGGlE_CONFIG_DIR to the mount path, but that didn't work:
os.environ["KAGGLE_CONFIG_DIR"] = "/.kaggle"
Then I tried to write the key to /tmp, and set KAGGlE_CONFIG_DIR to /tmp and that didn't work either.
def get_kaggle_api_key():
kaggle_key_path = "/.kaggle/kaggle.json"
#read the kaggle key
with open(kaggle_key_path, 'r') as f:
kaggle_api_key = f.read().strip()
#save the key to /tmp/
kaggle_config_path = "/tmp/kaggle.json"
with open(kaggle_config_path, 'w') as f:
json.dump(json.loads(kaggle_api_key), f)
os.environ["KAGGLE_CONFIG_DIR"] = "/tmp"
I always get this error:
OSError: Could not find kaggle.json. Make sure it's located in /www-data-home/.config/kaggle. Or use the environment method. See setup instructions at /
The setup instructions don't help, they just say to set KAGGLE_CONFIG_DIR to where the key is stored, which I did.
This is my first time trying to write a cloud function to access an api, and I'm not really sure what I'm doing. Please help!
I'm trying to write a google cloud function that automatically downloads data from a certain kaggle data set and upload it to a google cloud bucket. No matter how I try to include my kaggle api key in my google cloud function, the kaggle api can't find it!
I uploaded my kaggle api key to the secret manager and I wrote some code to write the key value into a file stored in /tmp, and set KAGGLE_CONFIG_DIR to /tmp. I granted the cloud function secret manager admin permissions and created added a runtime service account with permission to access the kaggle key as well just to make sure
I reference the secret via a mount path that I set here
First I tried just setting KAGGlE_CONFIG_DIR to the mount path, but that didn't work:
os.environ["KAGGLE_CONFIG_DIR"] = "/.kaggle"
Then I tried to write the key to /tmp, and set KAGGlE_CONFIG_DIR to /tmp and that didn't work either.
def get_kaggle_api_key():
kaggle_key_path = "/.kaggle/kaggle.json"
#read the kaggle key
with open(kaggle_key_path, 'r') as f:
kaggle_api_key = f.read().strip()
#save the key to /tmp/
kaggle_config_path = "/tmp/kaggle.json"
with open(kaggle_config_path, 'w') as f:
json.dump(json.loads(kaggle_api_key), f)
os.environ["KAGGLE_CONFIG_DIR"] = "/tmp"
I always get this error:
OSError: Could not find kaggle.json. Make sure it's located in /www-data-home/.config/kaggle. Or use the environment method. See setup instructions at https://github/Kaggle/kaggle-api/
The setup instructions don't help, they just say to set KAGGLE_CONFIG_DIR to where the key is stored, which I did.
This is my first time trying to write a cloud function to access an api, and I'm not really sure what I'm doing. Please help!
Share Improve this question edited Nov 15, 2024 at 22:04 Doug Stevenson 319k36 gold badges456 silver badges473 bronze badges asked Nov 15, 2024 at 21:58 Sean MacInnisSean MacInnis 11 silver badge1 Answer
Reset to default 1I'm not a Kaggle user but hopefully this example will help.
There are 3 (possibly 2) steps but importantly you must create a volume and then create a volume mount:
- Create an environment variable for
KAGGLE_CONFIG_DIR - Create a Volume (e.g.
KAGGLE-CONFIG-DIR) using the Secret - Create a Volume Mount (e.g.
/.kaggle)
Set an environment variable (KAGGLE_CONFIG_DIR) to /.kaggle:
Create a Volume:
Create a Container Volume Mount:
And some Python:
@functions_framework.http
def root(request):
# Get the value from the environment
kaggle_config_dir = os.getenv("KAGGLE_CONFIG_DIR")
# Open the file
with open(f"{kaggle_config_dir}/kaggle.json", 'r') as file:
content = file.read()
# Return the content
return f"{kaggle_config_dir}/kaggle.json: {content}:
Yields:
.kaggle/kaggle.json: {"your":"JSON"}