How to use my own modified version of codeql core library

[lllssskkk]

I'm modifying how tainted label propagate for javascript.

However, running codeql database analyze with -search-path flag couldn't instruct that codeql engine to pick up my version of code.

I have a file structure like the following

• codeql-runner
  • run_codeql_queries.py
• queries
  • FlowState
    • NewTaintedTrack.ql
  • OldTaintedTracking.ql
  • codeql-pack.lock.yml
  • qlpack.yml
  • queries.xml
• modified-codeql

The modified-codeql is a fork of this github repo https://github.com/yuske/codeql

The run_codeql_queries.py invokes following commands

system(f"cd ./queries && codeql pack install && cd -")
system(f"codeql database analyze {db} {./queries/OldTaintedTrack.ql} --search-path={absolute-path-to-modified-codeql} --output={path-to-sarif})

How does i know the codeql engine didn't use modified-codeql ?

CodeQL starts deprecating old version of tainted tracking API. NewTaintedTrack.ql uses the DataFlow::StateConfigSig which is not defined in the repo. The repo is from 2 years ago. Ideally, commands in my python script should fail as a result.

[aibaars]

Have you tried the --additional-packs= flag? What is the contents of your qlpack.yml and codeql-pack.lock.yml files? Are you using a CodeQL CLI or a "bundle" downloaded from github/codeql-action? The "bundle" contains its own copy of the standard libraries.

The easiest way to get things to work would be to put your query in modified-codeql/javascript/ql/src and run it from there.

[lllssskkk]

Hi, thanks for getting back to me.

I also tried --additional-packs={absolute-path-to-modified-codeql} flag.

system(f"cd ./queries && codeql pack install && cd -")
system(f"codeql database analyze {db} {./queries/FlowState/OldTaintedTrack.ql} --additional-packs={absolute-path-to-modified-codeql} --output={path-to-sarif})

It still runs smoothly.

I'm using the bundle.

Content of qlpack.yml

name: placeholder
version: 0.0.0
dependencies: 
  codeql/javascript-all: "2.0.1"

Content of ocdeql-pack.lock.yml

---
lockVersion: 1.0.0
dependencies: {}
compiled: false

I tried to put my query inside modified-codeql/javascript/ql/src, NewTaintedTrack.ql still runs.

The following is the log for codeql database analyze {db_path} {./modified-codeql/javascript/ql/src/queries/FlowState/NewTaintedTrack.ql} --additional-packs={absolute-path-to-modified-codeql} --format=sarif-latest

WARNING: Pack 'codeql/javascript-all' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/javascript-all/2.3.0/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/dataflow' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/dataflow/1.1.9/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/mad' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/mad/1.0.15/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/regex' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/regex/1.0.15/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/ssa' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/ssa/1.0.15/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/threat-models' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/threat-models/1.0.15/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/tutorial' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/tutorial/1.0.15/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/typetracking' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/typetracking/1.0.15/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/util' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/util/2.0.2/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/xml' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/xml/1.0.15/qlpack.yml:1,1-1)
WARNING: Pack 'codeql/yaml' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (/usr/local/codeql/codeql/qlpacks/codeql/yaml/1.0.15/qlpack.yml:1,1-1)
WARNING: To avoid these warnings in the future, use the '--no-strict-mode' option.
Dependencies resolved. Installing packages...
Install location: /root/.codeql/packages
Nothing to install.
Package install location: /root/.codeql/packages
Nothing downloaded.

New Test :

Removing /usr/local/codeql/codeql/qlpacks/ from the filesystem could make query not using the pre-compiled packs. However, the query is still trying to fetch the latest library from somewhere(maybe online?), not building from local libraries.

Dependencies resolved. Installing packages...
Install location: /root/.codeql/packages
Package install location: /root/.codeql/packages
Already installed codeql/dataflow@1.1.3 (library)
Already installed codeql/javascript-all@2.0.1 (library)
Already installed codeql/mad@1.0.9 (library)
Already installed codeql/regex@1.0.9 (library)