dsio is a command line tool for Google Cloud Datastore.
This tool is under development. Please use in your own risk.
- Bulk upsert entities from CSV and YAML file.
- Query by GQL from command line.
I have been developing web application which uses Datastore to store application's master data. I want to version-control master data in Datastore, and want to automatically reflect the changes to Datastore using CI service.
like below:
go get -u github.com/nshmura/dsio
- Create a service account.
- Set the following environment variable:
- DSIO_KEY_FILE : The path to the JSON key file.
- DSIO_PROJECT_ID : Developers Console project's ID (e.g. bamboo-shift-455)
Or execute dsio command with --key-file and --project-id options:
$ dsio upsert --key-file <path-to-service_account_file> --project-id <project-id> ...
If you want to connect to local Datastore emnulator, execute below command:
$(gcloud beta emulators datastore env-init)
For more information, please see this document.
To upsert entities from CSV file to Datastore:
(e.g. upsert into Book kind)
$ dsio upsert filename.csv -k Book
To upsert entities from YAML file to Datastore:
$ dsio upsert filename.yaml
To specify namespace:
$ dsio upsert filename.yaml -n development
To query by GQL:
$ dsio query 'SELECT * FROM Book LIMIT 2'
Output with CSV format:
$ dsio query 'SELECT * FROM Book LIMIT 2' -f csv
To specify namespace (e.g. production namespace):
$ dsio query 'SELECT * FROM Book LIMIT 2' -n production
CAUTION: In CSV (and TSV) format, information about types may be dropped in some case, and noindex value is removed.
So in some case, there is no way to restore exactly same entities in Datastore from the generated CSV.
$ dsio help upsert
NAME:
dsio upsert - Bulk-upsert entities into Datastore.
USAGE:
dsio upsert [command options] filename
OPTIONS:
--namespace value, -n value namespace of entities.
--kind value, -k value Name of destination kind.
--format value, -f value Format of input file. <yaml|csv|tcv>. (default: "yaml")
--dry-run Skip Datastore operations.
--batch-size value The number of entities per one multi upsert operation. batch-size should be smaller than 500. (default: 500)
--key-file value name of GCP service account file. [$DSIO_KEY_FILE]
--project-id value Project ID of GCP. [$DSIO_PROJECT_ID]
--verbose, -v Make the operation more talkative.
--no-color Disable color output.
$ dsio help query
NAME:
dsio query - Execute a query.
USAGE:
dsio query [command options] "[<gql_query>]"
OPTIONS:
--namespace value, -n value namespace of entities.
--output value, -o value Output filename. Entities are outputed into this file.
--format value, -f value Format of output. <yaml|csv|tcv>. (default: "yaml")
--style value, -s value Style of output. <scheme|direct|auto>. (default: "scheme")
--page-size value Number of entities to output at once. (default: 50)
--key-file value name of GCP service account file. [$DSIO_KEY_FILE]
--project-id value Project ID of GCP. [$DSIO_PROJECT_ID]
--verbose, -v Make the operation more talkative.
--no-color Disable color output.