Repository navigation
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The change still has an unresolved documentation typo and lacks DCA results or an assessment of changed alerts and regressions for the production model migration.
2 open issues
What changed in this PR
This PR converts Ruby library and test data extensions from YAML to JSON to improve CodeQL CLI loading performance, while updating the Ruby query pack to discover both formats.
Changes:
- Converted Ruby framework model extensions from
.model.ymlto.model.json. - Converted the Ruby dataflow test extension to
.ext.json. - Added JSON model patterns to
ruby/ql/lib/qlpack.yml.
| File | Description |
|---|---|
| ruby/ql/test/library-tests/dataflow/summaries/Summaries.ext.yml | Removes the YAML summaries test data extension. |
| ruby/ql/test/library-tests/dataflow/summaries/Summaries.ext.json | Adds the converted JSON summaries test data extension. |
| ruby/ql/lib/qlpack.yml | Adds discovery patterns for JSON model extensions while retaining YAML patterns. |
| ruby/ql/lib/codeql/ruby/frameworks/terrapin/model.yml | Removes the YAML Terrapin model. |
| ruby/ql/lib/codeql/ruby/frameworks/terrapin/model.json | Adds the converted JSON Terrapin model. |
| ruby/ql/lib/codeql/ruby/frameworks/stdlib/Shellwords.model.yml | Removes the YAML Shellwords model. |
| ruby/ql/lib/codeql/ruby/frameworks/stdlib/Shellwords.model.json | Adds the converted JSON Shellwords model. |
| ruby/ql/lib/codeql/ruby/frameworks/stdlib/Pathname.model.yml | Removes the YAML Pathname model. |
| ruby/ql/lib/codeql/ruby/frameworks/stdlib/Pathname.model.json | Adds the converted JSON Pathname model. |
| ruby/ql/lib/codeql/ruby/frameworks/Sqlite3.model.yml | Removes the YAML Sqlite3 model. |
| ruby/ql/lib/codeql/ruby/frameworks/Sqlite3.model.json | Adds the converted JSON Sqlite3 model. |
| ruby/ql/lib/codeql/ruby/frameworks/regexp/model.yml | Removes the YAML regexp barrier model. |
| ruby/ql/lib/codeql/ruby/frameworks/regexp/model.json | Adds the converted JSON regexp barrier model. |
| ruby/ql/lib/codeql/ruby/frameworks/Mysql2.model.yml | Removes the YAML Mysql2 model. |
| ruby/ql/lib/codeql/ruby/frameworks/Mysql2.model.json | Adds the converted JSON Mysql2 model. |
| ruby/ql/lib/codeql/ruby/frameworks/Json.model.yml | Removes the YAML JSON namespace model. |
| ruby/ql/lib/codeql/ruby/frameworks/Json.model.json | Adds the converted JSON namespace model. |
| ruby/ql/lib/codeql/ruby/frameworks/data/internal/empty.model.yml | Removes the YAML empty model definitions. |
| ruby/ql/lib/codeql/ruby/frameworks/data/internal/empty.model.json | Adds the converted JSON empty model definitions. |
| ruby/ql/lib/codeql/ruby/frameworks/core/Regexp.model.yml | Removes the YAML core Regexp model. |
| ruby/ql/lib/codeql/ruby/frameworks/core/Regexp.model.json | Adds the converted JSON core Regexp model. |
| ruby/ql/lib/codeql/ruby/frameworks/ActiveSupport.model.yml | Removes the YAML ActiveSupport model. |
| ruby/ql/lib/codeql/ruby/frameworks/ActiveSupport.model.json | Adds the converted JSON ActiveSupport model. |
| ruby/ql/lib/codeql/ruby/frameworks/ActiveStorage.model.yml | Removes the YAML ActiveStorage model. |
| ruby/ql/lib/codeql/ruby/frameworks/ActiveStorage.model.json | Adds the converted JSON ActiveStorage model. |
| ruby/ql/lib/codeql/ruby/frameworks/actiondispatch/internal/Mime.model.yml | Removes the YAML Mime model. |
| ruby/ql/lib/codeql/ruby/frameworks/actiondispatch/internal/Mime.model.json | Adds the converted JSON Mime model. |
Review effort: Lite (auto)
Comment on lines
+21
to
+22
| - codeql/ruby/frameworks/**/model.json | ||
| - codeql/ruby/frameworks/**/*.model.json |
| // It is possible to insert tainted data into `SafeBuffer` that won't get | ||
| // sanitized, and this taint is then propagated via most of the methods. | ||
| // | ||
| // TODO: SafeBuffer also reponds to all String methods. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CodeQL CLI 2.27.3 supports JSON data extensions (aka MaD models), and can load them significantly faster than YAML. This PR converts those in Ruby's library pack (first commit), along with
$TEST.ext.ymlfiles (second commit). The conversions were automated by an LLM-generated Python script (which I will add in a separate PR).Although JSON is preferred going forward,
ruby/ql/lib/qlpack.ymlnow has patterns that will pick up both.model.ymland.model.jsonfiles, so that any YAML files that do get checked in won't get silently dropped.To verify successful conversion in the first commit, I cherry-picked it onto #23081, which has a temporary test that enumerates all tuples from data extensions. The test still passes after conversion.