Repository navigation
Split long Bugzilla buglist references into per-bug references - #2512
NachiketPatil26 wants to merge 1 commit into
Conversation
Some advisories reference a buglist.cgi URL that lists dozens of bug ids and is longer than the 1024 characters allowed for a reference URL, so inserting the advisory references failed with a DataError. When such a URL is too long, store one show_bug.cgi reference per bug instead. Reference: aboutcode-org#2500 Signed-off-by: Nachiket <nachiketpatil2604@gmail.com>
sanchit955
left a comment
There was a problem hiding this comment.
I checked this out locally against PostgreSQL 16 (head 0f248c9). Both new tests pass, and the full vulnerabilities/tests/pipes/test_advisory.py file is 14/14 green. I also confirmed the new splits test fails on upstream main with the exact DataError: value too long for type character varying(1024) from #2500, so the test pins the fix. black --check is clean on both changed files.
One thing worth a thought for a follow-up: a long URL that is not a Bugzilla buglist still goes through unchanged, so it would still raise the same DataError and abort the advisory import. The Bugzilla case from the issue is covered — just flagging that the general case is still open.
kmlyz
left a comment
There was a problem hiding this comment.
split_long_bugzilla_buglist cleanly splits oversized /buglist.cgi URLs into show_bug.cgi?id=<bug_id> references for both Reference and ReferenceV2, while bulk_create(..., ignore_conflicts=True) safely handles duplicate IDs.
Fixes #2500
Some advisories (for example the Mozilla ones coming through NVD) reference a
buglist.cgi?bug_id=...URL that lists dozens of bugs and is longer than the 1024 charactersAdvisoryReference.urlallows, so inserting the references failed withvalue too long for type character varying(1024).get_or_create_advisory_referencesnow splits a Bugzilla buglist URL that is too long into oneshow_bug.cgi?id=Nreference per bug, keeping the reference type and settingreference_idto the bug id. Everything else, including short buglist URLs, is stored as before. Doing it there covers every importer.I went with splitting rather than a
TextFieldbecauseurlis part ofunique_togetherwithreference_type, so a long enough URL would still hit Postgres' index row size limit.Tests:
test_get_or_create_advisory_references_splits_long_bugzilla_buglistuses a 120-bug buglist URL; it fails on main with the sameDataErroras in the issue.test_get_or_create_advisory_references_keeps_short_bugzilla_buglistchecks short buglist URLs are left alone.pytest -m "not webtest": 919 passed, 1 skipped.