Skip to content

Commit dadea4f

Browse files
committed
feat(extract): Detect location of whosonfirst data from pelias.json
This change allows the Placeholder extract script to work in most cases _without_ specifying the `WOF_DIR` environment variable. Previously, unless you were using the particular arrangement of files and directories from pelias/docker, the default location the extract script looks for data (`/data/whosonfirst-data/sqlite`) was probably not correct. I noticed this inconvenience when running Pelias locally _without_ docker for the first time in quite a long time. My guess/recollection is an older version of the extract script (pre-sqlite) was pure bash, and so checking `pelias.json` was less convenient than in the current Node.js script. The `WOF_DIR` environment variable is left as an override, but my hope is with this change almost no one would have to use it.
1 parent 848ac1b commit dadea4f

File tree

2 files changed

+9
-3
lines changed

2 files changed

+9
-3
lines changed

README.md

+6-2
Original file line numberDiff line numberDiff line change
@@ -321,12 +321,16 @@ the database is created from geographic data sourced from the [whosonfirst](http
321321
322322
the whosonfirst project is distributed as geojson files, so in order to speed up development we first extract the relevant data in to a file: `data/wof.extract`.
323323
324-
the following command will iterate over all the `geojson` files under the `WOF_DIR` path, extracting the relevant properties in to the file `data/wof.extract`.
324+
the following command will iterate over all the `geojson` files downloaded by the Pelias whosonfirst importer, extracting the relevant properties in to the file `data/wof.extract`.
325325
326326
this process can take 30-60 minutes to run and consumes ~350MB of disk space, you will only need to run this command once, or when your local `whosonfirst-data` files are updated.
327327
328328
```bash
329-
$ WOF_DIR=/data/whosonfirst-data/data npm run extract
329+
$ npm run extract
330+
331+
# alternative if you do not have a `pelias.json` file specifying where WOF data should be
332+
$ WOF_DIR=/path/to/your/whosonfirst/data npm run extract
333+
330334
```
331335
332336
now you can rebuild the `data/store.json` file with the following command:

cmd/wof_extract_sqlite.js

+3-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,9 @@ const combinedStream = require('combined-stream');
99

1010
const SQLITE_REGEX = /whosonfirst-data-[a-z0-9-]+\.db$/;
1111

12-
const WOF_DIR = process.env.WOF_DIR || '/data/whosonfirst-data/sqlite';
12+
const WOF_DIR = process.env.WOF_DIR || // TODO: deprecate WOF_DIR env var after some time
13+
config.whosonfirst.datapath || // Preferred method of finding WOF data is to autodetect from pelias.json
14+
'/data/whosonfirst-data/sqlite'; // fallback to a hardcoded value. Correct for Pelias Docker images but likely nowhere else
1315

1416
const layers = fs.readFileSync(path.join(__dirname, 'placetype.filter'), 'utf-8')
1517
.replace(/^.*\(/, '') // Removes all characters before the first parenthesis

0 commit comments

Comments
 (0)