TOM (Talent Accelerator) Backend — Engineering Documentation

Audience: Backend developers joining or maintaining this codebase. Goal: Enable a new backend engineer to understand the architecture, business logic, data flow, and codebase well enough to be productive without a live knowledge-transfer session.

Method: Everything below is inferred from the actual source in this repository. Where a fact cannot be derived from the code, it is explicitly marked "Unable to determine from the codebase." Assumptions are labelled (Assumption). Facts are stated plainly.

---

Table of Contents

1. Backend Overview
2. Project Structure
3. Architecture
4. Module Breakdown
5. Request Lifecycle
6. Business Logic
7. API Layer
8. Database Interaction
9. Authentication & Authorization
10. Background Processing
11. Integrations
12. Configuration
13. Logging & Error Handling
14. Performance
15. Security
16. Dependency Map
17. Developer Guide
18. Code Quality Review
19. Improvement Opportunities
20. System Overview & Architecture
21. System Diagrams
22. API Reference
23. Version Control
24. Deployment & Maintenance

---

1. Backend Overview

Purpose

The TOM backend is the core engine of a compensation-management SaaS platform operated by The Talent Accelerator. It is a multi-tenant system where each tenant is a Company. For each company it manages the full lifecycle of compensation data and decisioning:

• Compensation reference data — job grades, salary ranges, cash allowances, short-term incentives (STI/bonus), long-term incentives (LTI/equity), benefit plans, internal payroll, and market survey data.
• Offer modelling — building individual candidate offers that pull from the reference data, computing a full compensation breakdown, generating offer emails/PDFs, and versioning revisions.
• Job evaluation & grading — scoring job roles against a fixed rules matrix and mapping the score to a company grade.
• Market benchmarking — the "Live Pay Benchmarking Tool" (LBT) that aggregates market pay statistics across companies.
• AI insights — LLM-generated compensation insights and analysis alerts on offers.
• Bulk data ingestion — CSV upload/download of all reference datasets, plus AI-assisted job-function mapping.

Main Responsibilities

Responsibility	Owning module(s)
Tenant & org configuration (companies, business units, regions, countries, legal entities, taxonomies)	tom_company_setup
Users, roles, permissions, JWT auth	authentication
Compensation reference data (grades, salary, STI, LTI, benefits, payroll, market data)	tom_compensation
Candidate offer modelling & compensation formula	tom_offer_modeler, services/offer_formula.py
Job evaluation scoring	job_evaluations, job_grades
Grade↔rank normalization	tom_grade
Market benchmarking (LBT)	benchmarking_tools
CSV ingestion + AI mapping	csv_uploader
AI insights & analysis alerts	ai_insights, ai_module
Dashboards & aggregation	dashboard
Company-specific custom job taxonomy	custom_jobs
File uploads to S3	s3_bucket
Cross-cutting: response envelope, exceptions, shared services, admin	response, tom_exceptions, services, utils, tom_admin, tom_applications, tom_enums, swagger

High-Level Architecture

flowchart TB
    Client[Frontend SPA / API Clients]
    subgraph Edge
        WN[WhiteNoise static]
        CORS[CORS Middleware]
        PROM[Prometheus Middleware]
    end
    subgraph Django["Django + DRF (Gunicorn)"]
        URLS[tombackend/urls.py]
        AUTH[JWT Auth + Permissions]
        VIEWS[App Views / Controllers]
        SVC[Service Layer<br/>services/*, per-app services/]
        SER[Serializers / Validation]
        ORM[Django ORM]
    end
    DB[(PostgreSQL)]
    REDIS[(Redis<br/>Celery broker + cache)]
    CELERY[Celery Workers]
    S3[(AWS S3)]
    subgraph External
        OPENAI[OpenAI]
        GROQ[Groq]
        FOREX[FastForex / Currency API]
        STOCK[MarketStack Stock API]
        SMTP[Office365 SMTP]
    end

    Client --> Edge --> URLS --> AUTH --> VIEWS
    VIEWS --> SER --> SVC --> ORM --> DB
    VIEWS --> SVC
    SVC --> REDIS
    VIEWS -. enqueue .-> REDIS --> CELERY
    CELERY --> ORM
    CELERY --> S3
    CELERY --> SMTP
    SVC --> OPENAI
    SVC --> GROQ
    SVC --> FOREX
    SVC --> STOCK
    VIEWS --> S3

Main Technologies

Layer	Technology	Version (from requirements.txt / Readme.md)
Language	Python	3.8 (Dockerfile) / 3.9 (Pipfile) — see note below
Web framework	Django	3.2.5
API framework	Django REST Framework	3.12.4
Auth	djangorestframework-simplejwt	4.7.2
Async tasks	Celery	5.2.7
Broker / cache	Redis	4.3.4
Database	PostgreSQL	13+ (via psycopg2-binary 2.9.1)
DB driver wrapper	django-prometheus (django_prometheus.db.backends.postgresql)	2.3.1
Object storage	AWS S3 via boto3 1.19.12 + django-storages 1.12.3	—
Data processing	pandas 1.3.1, numpy 1.24.4, openpyxl 3.0.10	—
LLM / AI	openai 1.109.1, groq 0.33.0, langchain 0.2.17, langchain-openai, langchain-groq	—
PDF / image rendering	pdfkit 1.0.0, imgkit 1.2.3 (wrap wkhtmltopdf / wkhtmltoimage)	—
API docs	drf-yasg 1.20.0 (Swagger)	—
Static files	whitenoise 6.7.0	—
History	django-simple-history 3.6.0 (installed, not actively used — see §18)	—
Testing	pytest 8.3.3, pytest-django, factory-boy, Faker	—
Lint/type	flake8, pylint, mypy, prospector, isort, pre-commit	—

Note (Python version discrepancy): Dockerfile uses python:3.8-slim, Pipfile declares python_version = "3.9", and Readme.md says Python 3.8.5. Treat 3.8 as the runtime of record (that is what the container ships). (Assumption)

Libraries reference (grouped by purpose)

requirements.txt pins 162 packages (including transitive deps). The ones that matter for development, grouped by what they do:

Category	Library (version)	Used for
Web framework	Django==3.2.5	Core framework, ORM, admin
	djangorestframework==3.12.4	REST API (views, serializers, permissions)
	djangorestframework-simplejwt==4.7.2	JWT auth + token blacklist
	django-cors-headers==3.7.0	CORS handling
	drf-yasg==1.20.0	Swagger/OpenAPI docs at /swagger/
Database	psycopg2-binary==2.9.1	PostgreSQL driver
django-prometheus==2.3.1	DB backend wrapper + /metrics
Async / queue	celery==5.2.7	Background tasks (reports, AI export, email)
	redis==4.3.4	Celery broker + result backend + cache
	django-celery-email==3.0.0	Email-via-Celery integration
Storage / files	boto3==1.19.12	AWS S3 access
	django-storages==1.12.3	S3 as Django's default file storage
	whitenoise==6.7.0	Static file serving
Data processing	pandas==1.3.1	CSV parsing/validation
	numpy==1.24.4	Benchmarking statistics (percentiles)
	openpyxl==3.0.10	Excel handling
AI / LLM	openai==1.109.1	OpenAI (AI mapping, ai_module summaries)
	groq==0.33.0	Groq (ai_insights messages)
	langchain==0.2.17 + langchain-openai==0.1.25 + langchain-groq==0.1.10 + langchain-core + langchain-text-splitters	LLM orchestration & prompts
Documents	pdfkit==1.0.0	Offer PDF (wraps wkhtmltopdf)
	imgkit==1.2.3	Offer email image (wraps wkhtmltoimage)
Auth / crypto	PyJWT==2.9.0	JWT encode/decode
HTTP	requests==2.32.4	External API calls (currency, stock)
Model utilities	django-enumchoicefield==2.0.0	Enum-backed model fields
	django-multiselectfield==0.1.12	Multi-select fields (benchmarking)
	django-simple-history==3.6.0	Audit history (installed but disabled — see §18)
	django-extensions==3.2.3	Dev/management helpers
	environs==9.5.0	Env-var settings loading
Testing / quality	pytest==8.3.3, pytest-django, factory-boy, Faker, coverage	Tests + fixtures + coverage
	flake8, pylint, mypy, prospector, isort, pre-commit	Lint / type-check / format

Server (not in requirements.txt but used in deployment): gunicorn — the WSGI server (procfile). Full pinned list: see requirements.txt (note: it is UTF‑16 encoded).

---

2. Project Structure

The repository is a modular Django project: one project package (tombackend) plus ~20 Django "apps," each a folder registered in INSTALLED_APPS (tombackend/settings/base.py:45-81).

text backend-prod/ ├── tombackend/ # Project root: settings, urls, wsgi/asgi, celery │ └── settings/ # Split settings: base + per-env (local/dev/staging/prod/docker) ├── authentication/ # Users, roles, permissions, JWT login (v1 + v2) ├── tom_company_setup/ # Multi-tenant config: companies, BUs, regions, countries, taxonomies ├── tom_compensation/ # Compensation reference data (grades, salary, STI, LTI, benefits, ...) ├── tom_offer_modeler/ # Candidate offer modelling + compensation formula ├── job_evaluations/ # Job evaluation scoring workflow ├── job_grades/ # Grade-point-range ↔ company value mapping ├── tom_grade/ # Company ↔ TA-rank grade normalization ├── benchmarking_tools/ # Live Pay Benchmarking Tool (LBT) ├── csv_uploader/ # Bulk CSV ingestion + AI job-function mapping ├── custom_jobs/ # Company-specific custom job taxonomy ├── ai_insights/ # Groq-backed compensation insight messages ├── ai_module/ # OpenAI-backed offer analysis alerts ├── dashboard/ # JE & TOM/CI dashboard aggregation ├── tom_admin/ # TOM (internal staff) admin user management ├── tom_applications/ # "Application" entitlements (TOM / JE / LBT / ...) ├── tom_enums/ # Application-module enum endpoint ├── s3_bucket/ # S3 file-upload endpoints ├── emails/ # (empty — no Python modules) ├── services/ # SHARED cross-app service layer (biggest: services.py, offer_formula.py) ├── response/ # Uniform response envelope + pagination mixins ├── tom_exceptions/ # Exception hierarchy, error codes, DRF exception handler ├── utils/ # Enums (Roles, Permissions), constants, base enum, helpers ├── swagger/ # drf-yasg schema view + reusable swagger fields ├── benchmarking_tools/, ... # (see Module Breakdown) ├── codedeploy/ # AWS CodeDeploy lifecycle hooks (before/after install, start/stop) ├── docker/ # Docker env files ├── manage.py, conftest.py, pytest.ini, requirements.txt, Pipfile, Dockerfile, ...

Folder-by-folder

Only meaningful folders are described. Each app follows Django conventions (models.py, views.py, urls.py, serializers*.py, migrations/, sometimes services/, enums.py, validators.py).

tombackend/ — Project configuration root

• Purpose: Holds Django settings, root URL routing, WSGI/ASGI entrypoints, and the Celery app.
• Important files:
• settings/base.py — all shared settings (installed apps, middleware, DRF config, DB, JWT, Celery, email, AWS, throttling, logging).
• settings/__init__.py — environment selector: reads the env OS variable and dynamically imports tombackend.settings.<env> (raises if env unset).
• settings/{local,dev,staging,prod,docker}.py — per-environment overrides (ALLOWED_HOSTS, FRONTEND_BASE_URL, DB URL, email backend, CELERY_ALWAYS_EAGER in local).
• celery.py — Celery app bootstrap; autodiscover_tasks across all installed apps.
• model.py — BaseModel abstract base (id BigAutoField, created_at, updated_at). Every domain model inherits this.
• urls.py — includes every app's urls.py under /api/… and mounts admin, swagger, Prometheus.
• Why it exists: Central wiring; nothing here contains business logic.

authentication/ — Identity & access

• Purpose: Custom user model, roles, permissions, JWT login (two API versions), password reset, account lockout, throttling.
• Responsibilities: CustomUser (email-based), RoleModel/PermissionModel/PermissionRoleModel/UserRoleModel, per-application role assignment, all permission_classes used across the platform (permissions.py, ~1000 lines).
• Important files: models.py, permissions.py, views.py (v1), v2/views.py (v2 login/verify), services/role_service.py, throttle.py, utils.py (lockout backend), serializers/.
• Why it exists: Single source of truth for who a user is and what they may do; referenced by every other app.

tom_company_setup/ — Multi-tenant foundation

• Purpose: Defines the tenant (CompanyModel) and all reference/org data hung off it.
• Responsibilities: Global taxonomies (Sector→Industry→SubIndustry, JobFunction→JobSubFunction), geography/currency (Country, Currency), per-company org structure (BusinessUnit, Region, RegionCountry, RegionsBusinessUnit, LegalEntity), stock tracking, company-user provisioning, company-role listing.
• Dependencies: Imports authentication, services, tom_applications; reaches into tom_offer_modeler.models and tom_compensation.models for referential delete-guards.
• Why it exists: It is the coupling hub — nearly every other app imports CompanyModel/CountryModel/CurrencyModel from here.

tom_compensation/ — Compensation reference-data master

• Purpose: Stores structured, versioned compensation data per company.
• Responsibilities: ~55 models across job grades, salary ranges, cash allowances, STI, LTI, benefits, internal payroll, market data, company job-function taxonomy, and TA-mapping.
• Important files: models.py (3k+ lines), views.py (3126 lines), serializers.py (3506 lines), services/company_function_delete_guard.py.
• Why it exists: The read-source that offer modelling, benchmarking, dashboards, and AI all consume.

tom_offer_modeler/ — Offer modelling

• Purpose: Build individual candidate offers, compute compensation, produce email/PDF, version revisions.
• Important files: models.py (offer aggregate + component models), views.py (1547 lines), serializers.py (1449 lines), email.py, services/{offer_formula via services/, versioning, copy_service, comparison_service, allowance_data}.py.
• Why it exists: The primary user-facing "decision" workflow.

services/ — Shared service layer

• Purpose: Cross-app business logic and query helpers used everywhere.
• Important files: services.py (1337 lines — ~60 role-scoped getters like get_business_units_by_company_id_and_role), offer_formula.py (684 lines — the compensation formula engine), email_service.py (all transactional emails + the _send_email Celery task), role_metafield_service.py, validate_metafield.py.
• Why it exists: DRY home for logic that would otherwise be duplicated across apps (row-level scoping, offer math, email).

response/, tom_exceptions/, utils/, swagger/ — Cross-cutting infrastructure

• response/ — TomResponse.get_response(...) (the uniform JSON envelope), TomPagination, QueryParamSortMixin (sort/search), custom renderer.
• tom_exceptions/ — exception classes (TomException, TomValidationException, TomResourceNotFoundException, …), the errors.py error-code catalog, and custom_exception_handler (wired as DRF EXCEPTION_HANDLER).
• utils/ — Roles and Permissions enums, constants.py (role groupings, URLs), base_enum.py, small helpers.
• swagger/ — drf-yasg schema view + reusable manual parameters.

Other apps

• ai_insights/, ai_module/, dashboard/, benchmarking_tools/, csv_uploader/, job_evaluations/, job_grades/, tom_grade/, custom_jobs/, tom_admin/, tom_applications/, tom_enums/, s3_bucket/ — all detailed in §4 Module Breakdown.
• emails/ — empty (no .py modules). Transactional email lives in services/email_service.py.

---

3. Architecture

Architectural patterns actually implemented

flowchart LR
    subgraph "Per Request"
        V[View<br/>GenericAPIView] --> S[Serializer<br/>validation + create/update]
        V --> SL[Service functions<br/>services.py / per-app services]
        S --> SL
        SL --> ORM[Django ORM<br/>Active Record models]
    end

1. Modular Monolith (Django apps). A single deployable Django project split into cohesive apps by domain. This is the dominant, explicit pattern (INSTALLED_APPS). There are no microservices — Celery workers run the same codebase, not separate services.

2. Layered architecture (loosely). Requests flow View → Serializer → Service → ORM. The layering is conventional, not enforced:

3. Controllers = DRF GenericAPIView subclasses (the codebase does not use ViewSets or routers).
4. Validation + write logic lives largely in serializers (create()/update() methods do real work).
5. Business/query logic lives in a service layer: the shared services/services.py and per-app services/ packages.

6. Data access = Django ORM (Active Record). There is no separate Repository layer — models and querysets are the data layer.

7. Service Pattern — yes. Explicit service classes/functions: RoleService, TomApplicationService, RoleMetafieldService, CIDashboardService, LivePayBenchmarkingReportBuilder, OfferVersioningService, OfferCopyService, OfferComparisonService, JobEvaluationService, JobGradeEvaluatorService, CSV service classes, AI service classes, plus the ~60 functions in services/services.py.

8. Repository Pattern — no. Not implemented. Data access is done directly via Model.objects / querysets, often inside services or serializers.

9. MVC / MVT. Standard Django MVT, but templates are used only for offer email/PDF rendering — the app is API-first (JSON).

10. CQRS — no. Reads and writes use the same models. (Read serializers vs write serializers is a light separation, not CQRS.)

11. Event-Driven — partial. Not a message/event bus. Asynchrony is task-queue based (Celery over Redis) for: LBT report generation, AI mapping export, and email sending. A minimal Django signal exists (authentication/signals.py).

12. Versioned reference data (domain pattern). A signature design choice in tom_compensation: each dataset has a Version model + Data model + child scoping models. Exactly one version is active per company (UniqueConstraint(fields=["company","is_active"])), and "set-active" flips which version data rows are read through. See §8.

13. Soft-delete convention. Most models use is_active as a string sentinel: "TRUE" = active, None = deleted (there is no "FALSE"). Partial unique constraints (condition=Q(is_active="TRUE")) enforce uniqueness only among active rows. A few models (CompanyModel, tom_grade, ai_module) use a real BooleanField instead — this inconsistency is a known smell (§18).

Uniform response envelope

Every endpoint returns the same shape via TomResponse.get_response (response/resp.py):

json { "success": true, "is_validation_error": false, "message": "Human-readable message", "code": 0, "data": {}, "error": {}, "is_paginated": false, "pagination": {} }

---

4. Module Breakdown

Conventions used below: Entry points = URL prefix (all app URLs are included under /api/ in tombackend/urls.py). Permissions are DRF permission classes from authentication/permissions.py unless noted. Every model inherits BaseModel (id, created_at, updated_at).

4.1 authentication

• Purpose / business objective: Identity and access management — who a user is and what they can do, across multiple "applications" (TOM, JE, LBT).
• Entry points: /api/auth/ (v1), /api/v2/auth/ (v2).
• Models (models.py):
• PermissionModel (permissions) — a named permission.
• RoleModel (roles) — name, is_admin_role, is_active, company FK (null for TOM/global roles), metafield (JSON — holds row-level scoping id lists), M2M permissions through PermissionRoleModel.
• PermissionRoleModel (permission_roles) — role↔permission link with is_active.
• CustomUser (users) — email is the username (USERNAME_FIELD="email"), role FK (single "active" role), roles M2M through UserRoleModel, company FK, country FK, is_company_user, is_one_time_password, failed_login_attempts, locked_until, M2M applications. Rich helper methods: get_active_applications(), get_allowed_applications(), is_app_allowed(), register_failed_attempt(), reset_failed_attempts(), is_locked.
• UserRoleModel (unique_user_role) — per-application role assignment: user, role, application (enum), grant_access, metafield. This is how a company user gets different roles/permissions in TOM vs JE vs LBT.
• UserAuthModel (user_auths) — one-time-password / forgot-password tokens with expiry.
• PasswordResetAttempt (password_reset_attempts) — lockout counter for password reset.
• Controllers: views.py (v1 Login/Logout/RefreshToken/ForgotPassword/ChangePassword), v2/views.py (LoginView with claims tokens + VerifyUserView for per-application role switching).
• Services: services/role_service.py (RoleService — role create/update, per-app role provisioning, create_roles_on_company_create).
• Serializers: serializers/{serializers,user_serializers,role_serializers}.py.
• Middleware/backends: utils.py → LockoutAuthenticationBackend (the configured AUTHENTICATION_BACKENDS), raising AccountLocked when locked_until is in the future.
• Validation: validators.py (password + user field validators), Django AUTH_PASSWORD_VALIDATORS.
• Throttling: throttle.py — LoginThrottle (scope login, 10/min keyed by email) and ForgotPasswordThrottle (3 per 2 hours via cache).
• Dependencies: tom_applications, tom_company_setup (Company/Country), services, response, tom_exceptions, utils.
• Internal workflow: See §9.

4.2 tom_company_setup

• Purpose: The multi-tenant foundation — the CompanyModel tenant plus all reference/org data and company-user provisioning.
• Entry points: /api/ (e.g. sector/, company/, company/<id>/business-unit/).
• Key models (models.py): SectorModel→IndustryModel→SubIndustryModel; JobFunctionModel→JobSubFunctionModel; CountryModel, CurrencyModel; CompanyModel (tenant); StockTracking; BusinessUnitModel, RegionModel, RegionsBusinessUnitModel, RegionCountryModel, LegalEntityModel.
• CompanyModel (companies): name, address, country_headquarter, financial_year, contract dates, logo URLs, country/currency FK, status (ACTIVE/IN_PROGRESS/EXPIRED enum), reporting quotas (max_grade_function_reports, max_individual_reports_percent, used_* counters), M2M applications, is_active (real Boolean here).
• Controllers: one large views.py (~2117 lines, ~35 GenericAPIView classes) following the Create / GetList / GetAll / Retrieve(GET/PUT/DELETE) shape.
• Services: relies on shared services/services.py (role-scoped getters), authentication.RoleService, tom_applications.TomApplicationService. A module-level helper _update_role_metafield_with_resource appends newly-created resource ids into the creating user's role.metafield (row-level scoping maintenance).
• Serializers: serializers/serializers.py (~1224 lines) + serializers/company_user_serializers.py. Notable: CreateOrUpdateCompanySerializer provisions company + COMPANY_SUPER_USER + roles + applications + stock data in one transaction.
• Permissions: IsAuthenticated, IsAdmin, IsAdminOrOwnCompany, plus fine-grained CanCreateSector, CanCreateCompany, CanCreateBusinessUnit, etc.
• Dependencies: authentication, services, tom_applications, and (delete-guard coupling) tom_offer_modeler, tom_compensation.
• Known issues: CountryView/CurrencyView have no permission classes (public); heavy N+1 in nested read serializers; magic currency id 138. See §18.

4.3 tom_compensation

• Purpose: Versioned compensation reference-data master (grades, salary, allowances, STI, LTI, benefits, payroll, market data, company job-function taxonomy + TA mapping).
• Entry points: /api/company/<company_id>/… (grades, salary-range, cash-allowance, short-term-incentive, long-term-incentive, benefit-plans, internal-payroll, market-data, company-job-function[-mapping]).
• Models: ~55 models, mostly in Version + Data + child-scoping triads. Representative: CompanyJobGradeVersionModel/CompanyJobGradeModel/CompanyJobGradeCountryModel; CompanySalaryRange*; CompanyCashAllowance*; CompanyShortTermIncentivePlanModel + ...Plan{Grade,Country,Function,SubFunction}Model + value CompanyShortTermIncentiveModel; LTI mirror; CompanyBenefitPlanModel + scoping; CompanyInternalPayrollModel (wide employee table); CompanyMarketDataModel (p25/p50/p75); CompanyJobFunctionModel/CompanySubJobFunctionModel; CompanyJobFunctionMappingModel/CompanySubJobFunctionMappingModel (company→TA); CompanyJobFunctionMappingAIExportModel (async AI export tracking).
• Controllers: views.py (3126 lines) — List (with TomPagination, QueryParamSortMixin), Create (@transaction.atomic), Retrieve, and ~9 identical set-active version PATCH views. BulkReplaceCompanyJFMappingsView does full replace-diff of mappings.
• Serializers: serializers.py (3506 lines, ~40 serializers) — nested writes for plan+scoping; grade↔country intersection validation (duplicated ~4×).
• Services: services/company_function_delete_guard.py (blocks delete if referenced by active sub-functions, offers, or job evaluations).
• Permissions: [IsAuthenticated, IsAdminOrOwnCompany, Can<Verb><Dataset>].
• Dependencies: tom_company_setup (incl. its private _update_role_metafield_with_resource), authentication, services, tom_grade, csv_uploader.services, and (delete-guard) job_evaluations, tom_offer_modeler.

4.4 tom_offer_modeler

• Purpose: Model individual candidate offers and compute compensation.
• Entry points: /api/company/<company_id>/offer[s]/….
• Models: OfferPositionDetailModel, OfferCandidateDetailModel, CompanyEmailTemplate (O2O Company); component headers + child rows — OfferFixedCashModel/OfferFixedCashAllowanceModel, OfferSTIModel/OfferSTIBonusModel, OfferLTIModel/OfferOtherLTIModel, OfferBenefitModel/OfferBenefitOtherModel, OfferSignOnBonusModel/OfferSignOnOtherBonusModel; the OfferModel aggregate (status DRAFTED/PLACED/ACCEPTED/REJECTED/EDITED, stage POSITION_DETAILS/CANDIDATE_DETAILS/OFFER_MODELLER, self-FK original_offer for revisions, allowance_data_fingerprint); OfferVersionHistory.
• Controllers: views.py (1547 lines) — list (full + light), retrieve/patch/delete, staged create (position → candidate → modeller), currency conversion (v1/v2), comparator-data, auto-populate-data, allowance-data-status, email page/PDF, email template + S3 logo upload, revise, compare, range-type.
• Services: the compensation formula in services/offer_formula.py; plus services/{versioning,copy_service,comparison_service,allowance_data}.py; email.py (email/PDF rendering).
• Serializers: serializers.py (1449 lines) — resolves company job/sub function → TA via mapping tables; the big CreateOrUpdateOfferModellerSerializer builds all components.
• Permissions: [IsAuthenticated, IsAdminOrOwnCompany, Can…Offer].
• Dependencies: tom_compensation, tom_company_setup, authentication, services, tom_grade; external requests, imgkit, pdfkit, boto3.
• Business logic detail: see §6.2.

4.5 job_evaluations

• Purpose: Score a job role against a fixed rules matrix and map the score to a company grade.
• Entry points: /api/company/<company_id>/job-evaluations/… and /api/job-evaluations/enums/.
• Models: JobEvaluation (single model) — company scoping, TA + company function FKs, five "division" enum fields (Knowledge & Skills, Problem Solving, Stakeholder Mgmt, Decision Impact, Financial/Non-financial Responsibility), evaluation_model (financial/non_financial), evaluation_result (resolved grade value), status (open/evaluated/close), documents.
• Controllers: views.py — list (role-scoped), sort, list-with-completion, retrieve/update/delete, evaluate, submit, document upload, search.
• Services: services/job_evaluation.py (JobEvaluationService), services/job_grade_evaluator.py (JobGradeEvaluatorService.evaluate_grade — sums points from je_rules.py and maps to a JobGradeModel band).
• Serializers: create (all function-mapping validated), update, EvaluateJobSerializer (all divisions required), retrieve, list (completion %).
• Permissions: [IsAuthenticated, IsAdminOrOwnCompany, CanCreate/View/Update/DeleteJobEvaluation].
• Dependencies: tom_company_setup, tom_compensation (company functions), job_grades, authentication, services, tom_grade.
• Business logic detail: see §6.3.
• Known bugs: submit() writes non-existent submitted_by/submitted_at; JeDocsUploadView has no permission classes. See §18.

4.6 job_grades

• Purpose: Map a company to 25 fixed numeric "grade point ranges" (100–1000) and assign a company-specific value/label to each.
• Entry points: /api/job-grade-mapping/….
• Models: JobGradeModel (job_grades) — company FK (CASCADE), grade_point_range (choices from GradePointRange enum), grade_point_value (label), created_by. Unique (company, grade_point_range). Properties start_range/end_range.
• Controllers: views.py — list unassigned ranges, list all ranges w/ assignment, mapping list/create/update, all-companies, grade-points pivot, delete.
• Permissions: uniformly [IsAuthenticated, IsAdmin] (admin-only; company users blocked).
• Dependencies: tom_company_setup, authentication, tom_grade (serializer reuse).
• Known issues: delete by pk hard-deletes all companies' rows sharing that range; update conflict-validation is dead code. See §18.

4.7 tom_grade

• Purpose: Normalize each company's internal grade labels onto a canonical TA rank ladder.
• Entry points: /api/grade/….
• Models: TARankModel (ta_ranks, ordered rank ladder), GradeCompanyModel (grade_companies, links a company in), GradeCompanyRankModel (grade_company_ranks, the company's label per TA rank).
• Controllers: views.py — TA ranks, grade-company GET/POST, client-companies, company/all (auto-provisions mappings), update/delete.
• Permissions: [IsAuthenticated, Can…GradeCompanyMapping].
• Known issues: GetGradeCompanyView is a side-effecting GET (backfills data on read). See §18.

4.8 benchmarking_tools

• Purpose: The Live Pay Benchmarking Tool (LBT) — aggregate market pay statistics from offers + payroll and produce a CSV report.
• Entry points: /api/company/<company_id>/benchmarking-report/live-pay/…, custom-filtered/, function/, benchmarking-report-title/.
• Models: LivePayBenchmarkingReportModel (config + result, M2M scoping, percentiles, aging, stage, report_url), LivePayBenchmarkingReportTrackModel (snapshot), LivePayBenchmarkingReportLogModel (by-title audit).
• Controllers: views.py — create+trigger async report, list, download (S3 proxy), synchronous by-title lookup, custom peer companies, functions-with-data.
• Services: services/calculations.py (the numpy stats engine — percentiles, suppression thresholds, org- vs incumbent-weighting, aging), services/report_creation.py (LivePayBenchmarkingReportBuilder, 1082 lines — build + CSV + S3 upload), services/live_pay/{grade_service,title_service,by_title_service}.py, services/report_utils/*.
• Background: tasks.py → generate_live_pay_report (Celery).
• Permissions: [IsAuthenticated, IsAdminOrOwnCompany, Can…LBTReport]; custom-filtered/ and function/ are [IsAuthenticated] only (weak scoping — §18).
• Business logic detail: see §6.4.

4.9 csv_uploader

• Purpose: Bulk CSV ingestion/export for all reference datasets, plus AI-assisted company→TA job-function mapping.
• Entry points: /api/…/upload/ and /…/download/ per dataset; …/ai-suggestions/….
• Models: none (writes models owned by tom_compensation/tom_company_setup).
• Controllers: views.py — one Uploader + one Download APIView per dataset; ReUploadView.
• Services: services/csv_uploader.py (base validation toolkit), services/admin_csv.py, services/client_csv.py (2896 lines — one service per dataset, versioned bulk persistence), services/csv_services/{internal_payroll,market_data}.py, services/mapping_ai_service.py (1248 lines — OpenAI embedding + LLM mapping pipeline), services/mapping_ai_export_service.py.
• Background: tasks.py → generate_company_jf_mapping_ai_export (Celery, S3, fingerprint idempotency).
• Permissions: per-dataset Can… classes; ReUploadView is fully open (§15/§18).
• Business logic detail: see §6.5 and §6.6.

4.10 ai_insights

• Purpose: Groq-LLM short compensation-insight messages for an offer (pay trend, one-time payment, retention risk, roles in demand), with hardcoded fallbacks.
• Entry points: /api/company/<company_id>/offer/<offer_id>/{base_pay_insights,ttc_pay_insights,ai_insights,one_time_payment,retention_risk,roles_in_demand}/.
• Models: AIInsightModel (read-back log). Note: ai_insights.utils.store_insight_log actually writes to ai_module.AIInsight — a cross-app coupling/naming hazard (§18).
• Controllers: views.py — validate → load offer → cache → insight_* in utils.py → generate via langchain_helper.AIInsightGenerator (Groq llama-3.3-70b-versatile) → store + cache.
• Permissions: [IsAuthenticated, IsAdminOrOwnCompany, CanCreateOffer].

4.11 ai_module

• Purpose: OpenAI-backed rule-based offer analysis alerts (compa ratio, conversion rate, critical pay, retention risk, payment recommendation) refined into NL summaries.
• Entry points: /api/company/<company_id>/offer/<offer_id>/{compa-ratio-review,conversion-analysis,critical-pay-positioning,retention-risk,payment-recommendation,insights}/.
• Models: AIInsight (ai_insights table) — alert records.
• Controllers: views.py on AIViewMixin — validate → analyze → cache → generate OpenAI summary → dedupe+persist → respond. Cleaner/layered architecture than ai_insights.
• Services: utils.py (analyzers), rag_chain_utils.py (AISummaryGenerator, OpenAI gpt-4o-mini), rag_prompts.py, mixins.py, cache_utils.py, enums.py.
• Permissions: [IsAuthenticated, IsAdminOrOwnCompany, CanCreateOffer].
• Note: "RAG" naming is a misnomer — there is no retrieval, just prompt templating over analyzer output.

4.12 dashboard

• Purpose: Aggregate offer/payroll/salary/JE data into dashboard metrics + filter option lists for TOM/CI and JE dashboards.
• Entry points: /api/company/<company_id>/{dashboard-ci,dashboard-ci-filters,dashboard-filters,dashboard-data,dashboardje-filters,dashboardJE-data}/.
• Models: none.
• Services: service.py (CIDashboardService, dataclass filters, Decimal money math), utils.py (midpoint maps, compa-ratio/pay-gap calculators, currency conversion).
• Permissions: [IsAuthenticated, IsAdminOrOwnCompany, CanViewOffer] (+ CanViewTomDashboard/CanViewJeDashboard).

4.13 custom_jobs

• Purpose: Company-specific custom job-function/sub-function taxonomy (independent of the TA catalog).
• Entry points: /api/company/<company_id>/custom-jobs/…, /api/custom-sub-jobs/….
• Models: CustomJobFunctionModel (custom_job_functions), CustomJobSubFunctionModel (custom_job_sub_functions, with file_url JSON).
• Permissions: function endpoints [IsAuthenticated, IsAdminOrOwnCompany, Can…]; sub-function endpoints omit IsAdminOrOwnCompany and lack company scoping (§18).
• Known bugs: function delete signature drops company_id (broken); get misuses many=True. See §18.

4.14 tom_admin, tom_applications, tom_enums, s3_bucket, swagger, emails

• tom_admin — /api/admin/tom-user[s]/…. Internal TOM-staff (non-company) user CRUD. Views: TomAdminUserView, GetTomAdminUsersView, RetrieveTomAdminUserView. Permissions: Can…TomAdminUser.
• tom_applications — TomApplicationModel (tom_applications table) + TomApplicationsEnum (TOM/JE/LIVE_PAY_BENCHMARKING_TOOL/ADMIN_PANEL/SSO) + TomApplicationsMixin (gives CustomUser/CompanyModel their applications helpers) + TomApplicationService. No URLs of its own; drives entitlement logic.
• tom_enums — /api/tom_enums/tom-application-modules/ (returns application-module enum choices).
• s3_bucket — /api/upload-image/, /api/upload-docx/, /api/company/<id>/upload-policy/. Direct S3 uploads via boto3/django-storages. Includes DOCX sanitization.
• swagger — drf-yasg schema view (login-protected at /swagger/) + reusable manual parameters.
• emails — empty package.

---

5. Request Lifecycle

Middleware order (tombackend/settings/base.py:85-98)

A request passes top-to-bottom through this chain on the way in, and bottom-to-top on the way out.

flowchart TB
    REQ([Incoming request]) --> M1[corsheaders.CorsMiddleware<br/><i>CORS headers</i>]
    M1 --> M2[django_prometheus.PrometheusBeforeMiddleware<br/><i>start metrics timer</i>]
    M2 --> M3[SecurityMiddleware<br/><i>HTTPS / security headers</i>]
    M3 --> M4[SessionMiddleware<br/><i>load session</i>]
    M4 --> M5[CommonMiddleware]
    M5 --> M6[CsrfViewMiddleware<br/><i>CSRF protection</i>]
    M6 --> M7[AuthenticationMiddleware<br/><i>attach request.user</i>]
    M7 --> M8[MessageMiddleware]
    M8 --> M9[XFrameOptionsMiddleware<br/><i>clickjacking protection</i>]
    M9 --> M10[WhiteNoiseMiddleware<br/><i>serve static files</i>]
    M10 --> M11[PrometheusAfterMiddleware<br/><i>record metrics</i>]
    M11 --> VIEW([DRF view / URL routing])

End-to-end flow

sequenceDiagram
    participant C as Client
    participant MW as Middleware chain
    participant U as URLconf (tombackend/urls.py → app urls)
    participant A as JWTAuthentication
    participant P as Permission classes
    participant T as Throttle classes
    participant V as View (GenericAPIView)
    participant Se as Serializer
    participant Sv as Service layer
    participant O as ORM
    participant DB as PostgreSQL
    participant EH as custom_exception_handler

    C->>MW: HTTPS request (Authorization: Bearer <jwt>)
    MW->>U: routed request
    U->>A: resolve view, authenticate
    A->>A: decode JWT → request.user (or AnonymousUser)
    A->>P: has_permission(request, view)
    P-->>V: allowed? (else PermissionDenied)
    V->>T: allow_request?
    T-->>V: ok (else Throttled → 429)
    V->>Se: validate request data
    Se-->>V: validated_data (or ValidationError)
    V->>Sv: business logic (@transaction.atomic on writes)
    Sv->>O: queryset / save
    O->>DB: SQL
    DB-->>O: rows
    Sv-->>V: result
    V->>Se: serialize result
    V-->>C: TomResponse.get_response(...) JSON
    Note over EH: Any raised exception is caught by<br/>custom_exception_handler and mapped<br/>to the TomResponse envelope + status code

Key points: - Authentication is DRF-level (DEFAULT_AUTHENTICATION_CLASSES = JWT, Basic, Session) — the first that succeeds sets request.user. - Default permission is IsAuthenticated (DEFAULT_PERMISSION_CLASSES). Views add fine-grained classes on top. - Authorization happens in permission classes before the view body (see §9). - Write endpoints wrap logic in @transaction.atomic at the view or serializer level. - All exceptions funnel through tom_exceptions.exception_handler.custom_exception_handler (registered as DRF EXCEPTION_HANDLER), which normalizes them into the TomResponse envelope.

---

6. Business Logic

6.1 Multi-tenant company setup & provisioning

• What it does: Creates a company tenant together with its first COMPANY_SUPER_USER, its application entitlements, its default roles, and initial stock data.
• Where it starts: POST /api/company/ → CreateCompanyView.post (@transaction.atomic).
• Where the logic lives: tom_company_setup/serializers/serializers.py::CreateOrUpdateCompanySerializer → uses authentication.RoleService.create_roles_on_company_create, tom_applications.TomApplicationService, and MarketStack for international stock values. On success, a one-time password email is sent to the super user (services/email_service.py::send_one_time_password_email).
• Related models: CompanyModel, CustomUser, RoleModel/UserRoleModel, TomApplicationModel, StockTracking.
• Row-level scoping: as an admin creates BUs/regions/grades/functions, _update_role_metafield_with_resource appends the new ids into the creating user's role.metafield. The services/services.py getters later read these lists to constrain what each user can see (the platform's row-level authorization mechanism).

6.2 Offer modelling & the compensation formula

• What it does: Builds a candidate offer across three stages and computes the full compensation breakdown.
• Where it starts: POST /api/company/<id>/offer/ (position) → …/candidate-detail/ → …/offer-modeller/.
• Where the logic lives: validation/writes in tom_offer_modeler/serializers.py; the compensation math in services/offer_formula.py (get_offer_context_data_service orchestrates five component builders).

flowchart TB
    P[POST offer/ position-detail] -->|stage=POSITION/CANDIDATE| CD[POST candidate-detail]
    CD -->|stage=OFFER_MODELLER| AP[GET auto-populate-data<br/>pull reference comp + fingerprint]
    AP --> CM[GET comparator-data<br/>internal + external benchmarks]
    CM --> OM[POST offer-modeller<br/>save all components]
    OM -->|is_draft=false| PLACED[status=PLACED<br/>allowance_data_fingerprint stored]
    PLACED --> EMAIL[email-page / pdf]
    PLACED --> STATUS[PATCH status ACCEPTED/REJECTED]
    PLACED --> REV[POST revise → OfferCopyService deep-copy<br/>original=EDITED, new revision]
    REV --> CMP[GET compare versions]

Compensation formula (the heart of the platform), from services/offer_formula.py: - Fixed cash: monthly = annual/12; compa ratio = proposed monthly base ÷ internal salary-range mid; market ratio = proposed monthly base ÷ internal-payroll average base; allowances paired current vs proposed. - STI (bonus): bonus target amount = annual base × (bonus target % / 100); proposed sales incentive = (proposed_annual_base / proposed_base_pay_percentage) × (100 − percentage); Target Total Cash (TTC) = fixed total + total bonus (adjusted when a sales incentive is proposed). - LTI (equity): unvested + new-hire equity + annual grant; other-LTI split into unit vs non-unit bonuses. - Sign-on: one-time payment current/proposed totals. - Benefits: each computed by calculation_basis (percent_basic → % of base; percent_guaranteed/percent_actual → % of guaranteed cash; flat → value); respects include_in_total_remuneration / include_in_offer_detail. - Totals: total comp and total remuneration current/proposed with % differences. - Data-freshness: services/allowance_data.py::compute_allowance_data_fingerprint SHA-256-hashes the (id, updated_at) of the reference querysets an offer depends on; the retrieve endpoint compares the stored fingerprint to a fresh one to flag stale offers. - Revisions: OfferVersioningService generates version codes; OfferCopyService.copy_offer deep-copies an offer into a new revision (original → EDITED); OfferComparisonService diffs versions.

⚠️ There are three implementations of the compensation math — services/offer_formula.py, tom_offer_modeler/email.py, and comparison_service.py — which can drift. See §18.

6.3 Job evaluation scoring

• What it does: Sums points across five "divisions" from a static rules matrix and maps the total to a company grade band.
• Where it starts: POST /api/company/<id>/job-evaluations/<pk>/evaluate/.
• Where the logic lives: job_evaluations/services/job_grade_evaluator.py::JobGradeEvaluatorService.evaluate_grade using the ~575-line matrix in je_rules.py; then finds the JobGradeModel whose start_range ≤ points ≤ end_range and returns its grade_point_value (stored to evaluation_result). Lifecycle: open → evaluated → close (submit).
• Risk: matrix lookups use direct dict[key] indexing without fallback — a missing/invalid division value raises KeyError → 500 (mitigated by the evaluate serializer requiring all fields).

6.4 Live Pay Benchmarking (LBT)

• What it does: Aggregates market pay from offers + payroll into percentile/mean statistics, applies data-aging and small-sample suppression, generates a CSV report to S3, and emails the user.
• Where it starts: POST /api/company/<id>/benchmarking-report/live-pay/ → validates, checks data sufficiency, persists the report, and enqueues generate_live_pay_report (Celery).
• Where the logic lives: benchmarking_tools/services/calculations.py (numpy stats, suppression thresholds, org- vs incumbent-weighting, aging), services/report_creation.py::LivePayBenchmarkingReportBuilder (build + CSV + S3), services/live_pay/* (sufficiency + synchronous by-title lookup with quota enforcement).
• Statistical model: incumbent-weighted = stats over all raw records; organization-weighted = stats over each company's mean. Small-sample suppression thresholds (e.g. p50 needs ≥3 orgs/≥3 obs; p10/p90 need ≥5 orgs/≥10 obs) except for the requesting company's own data.

6.5 CSV ingestion

• What it does: Validates and bulk-loads reference datasets from CSV; produces downloadable templates/exports.
• Where it starts: POST /api/…/upload/ (multipart, field attachment, ?active= flag).
• Where the logic lives: csv_uploader/services/client_csv.py (per-dataset services). Flow: parse CSV to DataFrame (read_csv_inclusively, chardet encoding detection) → validate (nulls, duplicates, enums, numeric/date columns, countries/currencies/grades against DB and role metafields) → on error, email the errors and return (False, message); on success, create a new version, deactivate the old, bulk_create rows, and email success.

6.6 AI job-function mapping

• What it does: Suggests a mapping from a company's job functions/sub-functions to the TA catalog using OpenAI embeddings + LLM.
• Where it starts: synchronous GET …/ai-suggestions/download/ or async POST …/ai-suggestions/exports/ (Celery + S3, polled via …/exports/latest/).
• Where the logic lives: csv_uploader/services/mapping_ai_service.py — 4 phases: (1) embed TA + company parents/pairs (cached 7 days), (2) cosine-similarity scoring + greedy sub assignment, (3) LLM picks one TA parent per company function, (4) LLM maps subs within the locked parent; combined match % = harmonic mean; low-confidence rows blanked. Async export is idempotent via a SHA-256 source_fingerprint.

---

7. API Layer

Controller organization

• All controllers are DRF GenericAPIView subclasses (no ViewSets, no routers). One class per operation-group; HTTP verbs implemented as get/post/put/patch/delete methods.
• URL composition: tombackend/urls.py includes each app's urls.py. Most are mounted at /api/; exceptions: /api/auth/ (v1 auth), /api/v2/auth/ (v2 auth), /api/admin/ (tom_admin), /api/grade/ (tom_grade), /api/tom_enums/.

API versioning

• Only authentication is versioned — /api/auth/ (v1) and /api/v2/auth/ (v2, adds embedded claims tokens + per-application role verification). Everything else is unversioned. Two ad-hoc "v2" endpoints exist inside tom_offer_modeler (currency …/v2) but that is a per-route suffix, not a version namespace.

Request validation

• Primarily via DRF serializers; write serializers implement validate(), create(), update(). Some validation lives in view bodies (reading request.data/query_params directly). CSV validation is custom pandas logic in the CSV service classes.

Authentication & Authorization

• JWT bearer tokens (see §9). Authorization via stacked permission classes.

Error handling & response format

• Uniform envelope from TomResponse.get_response. All errors normalized by custom_exception_handler (see §13).

Major endpoints by module

All paths below are relative to the host; prefixes as noted. <cid> = company_id.

Auth (/api/auth/, /api/v2/auth/) | Method | Path | Purpose | |---|---|---| | POST | /api/auth/login/ | v1 login (?is_company=) | | POST | /api/auth/logout/ | Blacklist refresh token | | POST | /api/auth/refresh-token/ | Refresh access token | | POST | /api/auth/forgot-password/ | Send reset link | | POST | /api/auth/change-password/ | Reset via token | | POST | /api/v2/auth/login/ | v2 login (claims + application) | | GET | /api/v2/auth/verify-user/ | Switch/verify per-application role |

TOM Admin (/api/admin/): tom-user/ (POST), tom-users/ (GET), tom-user/<pk>/ (GET/PUT/PATCH).

Company setup (/api/): permissions/, sector[s]/, industry/industries/, sub-industry/…, job-function[s]/, job-sub-function[s]/, countries/, currencies/, company/, companies[/all|/list]/, company/<cid>/, company/<cid>/business-unit[s]/…, …/region[s]/…, …/legal-entity/…, …/user[s]/…, …/roles/…, …/stocks/update/.

Compensation (/api/company/<cid>/): job-grade[s]/…, salary-range[s]/…, cash-allowance[s]/…, short-term-incentive[-plan][s]/…, long-term-incentive[-plan][s]/…, benefit-plans/…, internal-payroll-versions/…, market-data-versions/…, company-job-function[s]/…, company-sub-job-function[s]/…, company-job-function-mapping[s]/…. Each dataset: …-versions/ (GET), …-version/<pk>/set-active/ (PATCH), list (GET), create (POST), <pk>/ (GET/PUT/DELETE).

Offer modeler (/api/company/<cid>/): offers[/list]/, offer/<pk>/, offer/ (POST), offer/<pk>/{position-detail,candidate-detail,offer-modeller,comparator-data,auto-populate-data,allowance-data-status,email-page[/pdf],revise,compare}/, offer/<pk>/currency/<cur>[/v2], add-template/, range_type/.

Job evaluations (/api/): job-evaluations/enums/, company/<cid>/job-evaluations[/list|-sort]/, …/<pk>/, …/<pk>/{evaluate,submit,documents}/, …/search.

Job grades (/api/): job-grade-mapping/{,<pk>/delete/,grade-point-ranges[-assigned]/,grade-points/,company/all}.

TOM grade (/api/grade/): ta-ranks/, company/, client-companies/, company/all, company/<pk>/.

Benchmarking (/api/company/<cid>/): benchmarking-report/live-pay[/list]/, benchmarking-report-get/<pk>/, benchmarking-report-title/, custom-filtered/, function/.

CSV (/api/): {sector,job-function,custom-jobs,grade-table}/{upload,download}/, company/<cid>/{job-grade,salary-range,cash-allowance,short-term-incentive,long-term-incentive,benefits-plan,internal-payroll,market-data,company-job-function,company-job-function-mapping}/{upload,download}/, …/company-job-function-mapping/ai-suggestions/{download,exports[/latest]}/, upload/re-upload.

AI insights (/api/company/<cid>/offer/<offer_id>/): base_pay_insights/, ttc_pay_insights/, ai_insights/, one_time_payment/, retention_risk/, roles_in_demand/.

AI module (/api/company/<cid>/offer/<offer_id>/): compa-ratio-review/, conversion-analysis/, critical-pay-positioning/, retention-risk/, payment-recommendation/, insights/.

Dashboard (/api/company/<cid>/): dashboard-ci/, dashboard-ci-filters/, dashboard-filters/, dashboard-data/, dashboardje-filters/, dashboardJE-data/.

Custom jobs (/api/): company/<cid>/custom-jobs/{create,,all,<pk>}/, custom-sub-jobs/{create,,<pk>}/.

S3 (/api/): upload-image/, upload-docx/, company/<cid>/upload-policy/.

Enums (/api/tom_enums/): tom-application-modules/.

Ops: /metrics (django-prometheus), /swagger/ (login-protected), /admin/ (Django admin).

---

8. Database Interaction

ORM

• Django ORM exclusively (Active Record). The DB backend is django_prometheus.db.backends.postgresql (wraps PostgreSQL to expose query metrics).
• No raw SQL of significance and no Repository layer — queries are built inline in services/serializers/views.

Models & relationships

• Every model inherits tombackend.model.BaseModel (id BigAutoField, created_at, updated_at).
• AUTH_USER_MODEL = "authentication.CustomUser".
• The dominant relational shape in tom_compensation is the Version → Data → child-scoping triad:

erDiagram
    CompanyModel ||--o{ CompanyJobGradeVersionModel : has
    CompanyJobGradeVersionModel ||--o{ CompanyJobGradeModel : contains
    CompanyJobGradeModel ||--o{ CompanyJobGradeCountryModel : scoped_to
    CompanyModel ||--o{ CompanySalaryRangeVersionModel : has
    CompanySalaryRangeVersionModel ||--o{ CompanySalaryRangeModel : contains
    CompanyModel ||--o{ CompanyShortTermIncentivePlanModel : has
    CompanyShortTermIncentivePlanModel ||--o{ CompanyShortTermIncentiveModel : values
    OfferModel }o--|| CompanyModel : belongs_to
    OfferModel }o--|| OfferPositionDetailModel : has
    OfferModel }o--o| OfferCandidateDetailModel : has
    OfferModel }o--o| OfferFixedCashModel : has
    OfferModel }o--o| OfferSTIModel : has
    OfferModel }o--o| OfferLTIModel : has
    OfferModel }o--o| OfferBenefitModel : has
    OfferModel }o--o| OfferSignOnBonusModel : has
    CustomUser }o--|| RoleModel : active_role
    CustomUser }o--o{ RoleModel : roles_via_UserRoleModel
    RoleModel }o--o{ PermissionModel : via_PermissionRoleModel
    CompanyModel ||--o{ CustomUser : employs

Schema reference — data types & character limits

Types below are Django field types and their PostgreSQL equivalents. Every table also has the inherited BaseModel columns: id (BigAutoField → bigint, PK), created_at (DateTimeField → timestamptz), updated_at (DateTimeField → timestamptz). "Limit" = max_length (chars), max_digits/decimal_places (numeric precision), or the value range.

users (authentication.CustomUser)

Column	Django type	PostgreSQL type	Limit / precision	Null?	Notes
email	EmailField	varchar	254 (Django default)	No	unique; is the login (USERNAME_FIELD)
password	CharField	varchar	255, min 8	No	hashed
first_name / last_name	CharField	varchar	150 (Django default)	—	from AbstractUser
phone_number	CharField	varchar	20	Yes	unique with phone_code
phone_code	IntegerField	integer	−2.1B…2.1B	Yes
role_id	ForeignKey → roles	bigint	—	Yes	PROTECT
country_id	ForeignKey → countries	bigint	—	Yes	PROTECT
company_id	ForeignKey → companies	bigint	—	Yes	PROTECT
is_active, is_superuser, is_staff, is_company_user, is_one_time_password	BooleanField	boolean	—	No	default False (except is_active=True)
failed_login_attempts	PositiveIntegerField	integer	≥ 0	No	default 0
locked_until	DateTimeField	timestamptz	—	Yes	lockout expiry
date_joined	DateTimeField	timestamptz	—	No	default now

roles (RoleModel) & permissions (PermissionModel)

Table.Column	Type	PostgreSQL	Limit	Null?
roles.name	CharField	varchar	150	No
roles.description	CharField	varchar	255	Yes
roles.is_admin_role / is_active	BooleanField	boolean	—	No
roles.metafield	JSONField	jsonb	—	No
roles.company_id	FK → companies	bigint	—	Yes
permissions.name	CharField	varchar	150	No
permissions.description	CharField	varchar	255	Yes

user_auths (UserAuthModel) — password reset / OTP tokens

Column	Type	PostgreSQL	Limit	Null?
token	CharField	varchar	255	Yes
otp	IntegerField	integer	—	Yes
auth_type	CharField (choices)	varchar	255	No
expiry	DateTimeField	timestamptz	—	Yes
is_active	BooleanField	boolean	—	No

companies (CompanyModel)

Column	Django type	PostgreSQL	Limit	Null?
name	CharField	varchar	255, unique	No
address	TextField	text	unbounded	No
postal_code	CharField	varchar	12	Yes
country_headquarter	CharField	varchar	255	No
financial_year	PositiveIntegerField	integer	≥ 0	No
contract_start_date / contract_end_date	DateTimeField	timestamptz	—	No
large_logo_url / small_logo_url	TextField	text	unbounded	No
stock_tracking_ids	ArrayField(ArrayField(CharField))	varchar[][]	inner 255	No
country_id / currency_id	FK	bigint	—	No (PROTECT)
status	EnumChoiceField	varchar	ACTIVE / IN_PROGRESS / EXPIRED	No
max_grade_function_reports	IntegerField	integer	—	Yes (default 2)
max_individual_reports_percent	FloatField	double precision	—	Yes (default 10.0)
is_active	BooleanField	boolean	—	No

countries / currencies / soft-delete taxonomies

Table.Column	Type	PostgreSQL	Limit	Null?
countries.name	CharField	varchar	255, unique	No
countries.is_system_field	BooleanField	boolean	—	No
currencies.name	CharField	varchar	255, unique	No
currencies.symbol	CharField	varchar	255	No
currencies.code	CharField	varchar	255, unique	No
sectors/industries/…name	CharField	varchar	255	No
sectors/…is_active	CharField (choices)	varchar	4 ("TRUE"/null)	Yes
stock_data.stock_value	DecimalField	numeric	12,2 (max ±9,999,999,999.99)	—
stock_data.tracking_id	CharField	varchar	255, unique	—
stock_data.stock_type	CharField (choices)	varchar	50	—

Compensation money fields (representative)

Table.Column	Type	PostgreSQL	Limit / precision	Notes
company_salary_ranges.salary_min/mid/max	FloatField	double precision	IEEE-754 float	⚠️ float, not decimal — see note
company_long_term_incentives.equity_min/mid/max	FloatField	double precision	float
company_cash_allowances.value / STI value	FloatField	double precision	float
company_benefits_plans.employer_cost_value	DecimalField	numeric	(max_digits/decimal_places set on model)	decimal (exact)
company_benefits_plans.benefit_limit	DecimalField	numeric	(as defined)	decimal (exact)
company_internal_payrolls.annual_base_pay	FloatField	double precision	float, non-null
company_internal_payrolls.annual_* (bonus, LTI, TR, …)	FloatField	double precision	float, nullable	~15 numeric columns
company_market_data.*_p25/p50/p75	FloatField	double precision	float	annual_base_pay_p50 non-null
company_job_grades.grade	CharField	varchar	(see model)

⚠️ Data-type caution for developers: most monetary values use FloatField (double precision), not DecimalField. Floats can introduce rounding error in money math (benefits use DecimalField, which is exact). This inconsistency is flagged in §18; when adding new money fields, prefer DecimalField(max_digits, decimal_places).

Offers (offers + component tables) — representative

Table.Column	Type	PostgreSQL	Limit	Null?
offers.status	CharField (choices)	varchar	40 (DRAFTED/PLACED/ACCEPTED/REJECTED/EDITED)	No
offers.stage	CharField (choices)	varchar	40	No
offers.version_code	CharField	varchar	50	Yes
offers.rejection_reason	TextField	text	unbounded	Yes
offers.allowance_data_fingerprint	CharField	varchar	64 (SHA-256 hex)	Yes
offer_position_details.grade / reporting_grade	CharField	varchar	30	No / Yes
offer_position_details.job_title / city / range_type	CharField	varchar	255	mixed
offer_candidate_details.candidate_name	CharField	varchar	255	No
offer_candidate_details.gender	CharField (choices)	varchar	20	Yes
offer_fixed_cash.*_annual_base, *_compa_ratio	FloatField	double precision	float	Yes

Full column-level schema for every table is the model files themselves (<app>/models.py) — this reference covers the high-traffic tables. To regenerate a complete schema, run python manage.py inspectdb or python manage.py sqlmigrate <app> <migration>.

The versioning lifecycle

stateDiagram-v2
    [*] --> NoVersion
    NoVersion --> ActiveV1: create version (auto-active if first)
    ActiveV1 --> ActiveV2: upload new CSV creates V2, deactivates V1
    ActiveV2 --> ActiveV1: set-active PATCH on V1
    note right of ActiveV2
        Exactly one version has is_active=TRUE
        per company (UniqueConstraint).
        Data rows are read "through" the active version.
    end note

Transactions

• Write endpoints use @transaction.atomic widely (view- or serializer-level; present across tom_compensation, tom_offer_modeler, csv_uploader, tom_company_setup, benchmarking_tools, job_evaluations, s3_bucket, etc.).
• OfferVersioningService uses select_for_update() to lock the root original offer when generating a revision code (prevents duplicate version codes under concurrency).
• Gap: several cascade-nulling soft-delete methods are not wrapped in atomic, risking partial deletes (§18).

Query flow & data lifecycle

• Reads go through role-scoped getters in services/services.py that filter by company_id and by the id-lists in the user's role.metafield (row-level scoping), then by the active version.
• Writes (create/update) go through serializer create()/update(), which resolve FKs, run cross-entity validation, and bulk_create child rows.
• Soft-delete flips is_active to None (cascading to children).

Performance concerns (DB)

• N+1 queries are pervasive — nested SerializerMethodFields re-query per parent row across nearly every app; list views generally lack select_related/prefetch_related. The LBT calculation path is the highest risk: querysets OR-combined in per-grade/per-company loops, then deep related access (offer.offer_fixed_cash, offer.offer_sti, …) with no prefetching.
• Counter updates use read-modify-write ((x or 0)+1) instead of F() expressions (race-prone).
• See §14 for the full list.

---

9. Authentication & Authorization

Login flow (v2 — the current path)

sequenceDiagram
    participant C as Client
    participant LV as v2 LoginView
    participant BE as LockoutAuthenticationBackend
    participant U as CustomUser
    C->>LV: POST /api/v2/auth/login/ {email, password, application?}
    LV->>LV: LoginThrottle (10/min per email)
    LV->>BE: auth.authenticate(email, password)
    alt account locked
        BE-->>LV: raise AccountLocked
        LV-->>C: 400 ACCOUNT_LOCKED (minutes remaining)
    else invalid
        BE->>U: register_failed_attempt() (lock after 5)
        LV-->>C: 400 INVALID_CREDENTIALS
    else valid
        BE->>U: reset_failed_attempts()
        alt is_one_time_password
            LV->>U: create ONE_TIME_PASSWORD UserAuthModel
            LV-->>C: {is_one_time_password:true, token}
        else normal
            LV->>LV: resolve per-application role (UserRoleModel)
            LV->>LV: RefreshToken.for_user + ClaimsToken(role, permissions)
            LV-->>C: {user, token{access,refresh,claims}, applications}
        end
    end

Tokens

• JWT via djangorestframework-simplejwt. Access token lifetime 1 day, refresh 5 days (SIMPLE_JWT in base settings).
• v2 additionally issues a ClaimsToken (authentication/tokens.py) embedding role and permissions.
• Logout blacklists the refresh token (rest_framework_simplejwt.token_blacklist is installed).
• Sessions are also enabled (Basic + Session auth are in the default classes), used for Django admin and Swagger.

Password reset & one-time password

• forgot-password/ issues a UserAuthModel token (5-min expiry), emails a reset link, and is throttled (3 per 2h per email). New company users get a one-time password; on first login they receive a reset token and must set a new password (change-password/).

Account lockout

• LockoutAuthenticationBackend (the sole AUTHENTICATION_BACKENDS entry) locks a user for 15 minutes after 5 failed attempts (CustomUser.register_failed_attempt).

Permission system

flowchart TB
    subgraph Identity
        U[CustomUser] -->|active role| R[RoleModel]
        U -->|per-application| UR[UserRoleModel: app → role + metafield]
        R -->|M2M via PermissionRoleModel| PM[PermissionModel]
        R -->|metafield JSON| MF[Row-level scope:<br/>business_units, regions,<br/>countries, grades, job_functions]
    end
    subgraph Enforcement
        V[View permission_classes] --> HP["_has_permission(request, perm, methods, is_company)"]
        HP -->|TOM_MASTER_USER| ALLOW[allow]
        HP -->|is_company & COMPANY_SUPER_USER| ALLOW
        HP -->|method not in class methods| ALLOW
        HP -->|role has permission| ALLOW
        HP -->|else| DENY[deny 403]
    end
    subgraph RowLevel
        SVC[services.services getters] -->|read role.metafield| MF
        SVC --> QS[filtered querysets]
    end

Roles (utils/enums.py::Roles): TOM_MASTER_USER, TOM_SUPER_USER, TOM_ADMIN, TOM_SALES (internal staff); COMPANY_SUPER_USER, COMPANY_ADMIN, COMPANY_USER, COMPANY_BUSINESS_ACCESS (tenant users).

Two enforcement layers: 1. Object/verb permissions — DRF permission classes (authentication/permissions.py). The central helper _has_permission(request, permission, methods, is_company=False) returns True if the user is TOM_MASTER_USER, or (is_company and COMPANY_SUPER_USER), or the request method is not in the class's guarded method list, or the role holds the named permission. 2. Row-level scoping — the id-lists in role.metafield (business_units/regions/countries/grades/job_functions), consumed by services/services.py getters to filter querysets to what the user is allowed to see.

⚠️ Critical gotcha for reviewers: because _has_permission returns True when the request method isn't in a class's method list, stacking CanView + CanEdit + CanDelete on a Retrieve view means each verb is effectively gated only by its matching class. Do not assume "all listed permission classes must pass for every request." This also means a misconfigured method list silently allows access. See §15/§18.

Coarse permissions: IsAuthenticated (default), IsAdmin (rejects company users), IsAdminOrOwnCompany (admins, or company users whose company.id == view.kwargs["company_id"]).

---

10. Background Processing

Infrastructure

• Celery 5.2.7 over Redis (CELERY_BROKER_URL / CELERY_RESULT_BACKEND, default redis://localhost:6379/0). JSON serialization; timezone America/New_York.
• App defined in tombackend/celery.py; autodiscover_tasks scans all installed apps.
• In local settings, CELERY_ALWAYS_EAGER = True — tasks run synchronously (no worker needed for dev).
• In production, Celery runs under Supervisor alongside Gunicorn (see codedeploy/after_install.sh).

Tasks (there are exactly three producers)

flowchart LR
    subgraph Producers
        LBTV[LBT create view] -->|.delay| T1
        CSVV[AI export trigger view] -->|enqueue_export| T2
        ANY[Any email caller] -->|shared_task| T3
    end
    subgraph "Celery Worker"
        T1[generate_live_pay_report<br/>benchmarking_tools/tasks.py]
        T2[generate_company_jf_mapping_ai_export<br/>csv_uploader/tasks.py]
        T3[_send_email<br/>services/email_service.py]
    end
    T1 --> S3[(S3)]
    T1 --> MAIL[SMTP success email]
    T2 --> S3
    T3 --> SMTP[Office365 SMTP]

Task	Trigger	What it does	Retry / status
generate-live-pay-report (benchmarking_tools/tasks.py)	POST benchmarking-report/live-pay/ → .delay()	Marks report processing, runs LivePayBenchmarkingReportBuilder.build() + CSV, uploads to S3, marks completed, sends success email; on error marks error and re-raises.	Status via stage enum; no explicit max_retries.
generate-company-jf-mapping-ai-export (csv_uploader/tasks.py)	POST …/ai-suggestions/exports/ → enqueue_export	Marks export processing, generates AI mapping CSV, uploads to S3, stores file_url/row_count, marks completed.	bind=True, max_retries=3, default_retry_delay=60; MappingAIError is non-retryable, other exceptions self.retry. Idempotent via SHA-256 source_fingerprint.
_send_email (services/email_service.py)	Any transactional email helper	Sends via Django send_mail.	Decorated @shared_task (so can run async) — but most callers invoke the wrapper functions synchronously within the request (see §18).

Scheduled jobs / beat

• None active. tombackend/celery.py imports crontab but defines no beat_schedule. There is a manual management command authentication/management/commands/resend_otp_emails.py (run on demand, not scheduled). No Celery Beat schedule is configured in the codebase.

Event handlers

• One Django signal file (authentication/signals.py). simple_history's HistoryRequestMiddleware is commented out — no automatic history capture.

---

11. Integrations

flowchart LR
    APP[TOM Backend]
    APP -->|boto3 / django-storages| S3[(AWS S3<br/>files, reports, logos)]
    APP -->|langchain-openai: ChatOpenAI gpt-4o-mini + embeddings| OPENAI[OpenAI]
    APP -->|langchain-groq: llama-3.3-70b| GROQ[Groq]
    APP -->|requests GET| FOREX[FastForex / exconvert currency API]
    APP -->|requests GET| STOCK[MarketStack stock API]
    APP -->|SMTP TLS 587| O365[Office365 smtp.office365.com]
    APP -->|django-prometheus| PROM[Prometheus /metrics]

Integration	Purpose	Method	Where	Failure handling
AWS S3	Store uploaded files, LBT reports, offer logos, AI export CSVs	boto3 sessions + django-storages (DEFAULT_FILE_STORAGE = s3boto3)	s3_bucket/views.py, benchmarking_tools, csv_uploader, tom_offer_modeler	Exceptions propagate; report task marks error. Some inline credential use.
OpenAI	AI job-function mapping (embeddings + chat) and ai_module offer-analysis summaries	langchain_openai (ChatOpenAI gpt-4o-mini, OpenAIEmbeddings text-embedding-3-small)	csv_uploader/services/mapping_ai_service.py, ai_module/rag_chain_utils.py	Structured Pydantic output + heuristic fallback; ai_module returns None (graceful) if unavailable. Key env TOM_AI_PRIMARY_TOKEN.
Groq	ai_insights compensation-insight messages	langchain_groq.ChatGroq (llama-3.3-70b-versatile)	ai_insights/langchain_helper.py	On any exception → hardcoded backup messages (management/backup_message.py). Key env GROQ_API.
Currency API (FastForex / exconvert)	FX conversion for offers, dashboards, benchmarking	requests.get	dashboard/utils.py, services/offer_formula.py, services/services.py	Returns 1.0 on any failure (silent — risk of wrong numbers). Key CURRENCY_FAST_FOREX_API_KEY. No timeout/retry.
MarketStack (stock API)	Fetch stock prices for international stock tracking	requests.get	tom_company_setup/utils.py, model StockTracking.set_stock_value	Raises ValueError on failure (can surface inside a transaction). Key STOCK_API_KEY.
Office365 SMTP	All transactional email	Django send_mail (TLS, port 587)	services/email_service.py	_send_email re-raises; offer email send swallows exceptions (except: pass).
Prometheus	Metrics	django-prometheus middleware + DB backend	settings + /metrics	N/A

Note: requirements.txt also lists tiktoken, langsmith, sqlalchemy, httpx (transitive/optional AI-stack deps). No evidence of direct use beyond the LangChain integrations above.

---

12. Configuration

Environment selection

• tombackend/settings/__init__.py reads the env OS variable (local | dev | staging | prod | docker) and dynamically imports the matching settings module (raises ValueError if env is unset).
• Settings are loaded via environs.Env (.env file supported).

Environment variables (.env.example)

Variable	Purpose
SECRET_KEY	Django secret
env	Environment selector (local/dev/staging/prod)
DEBUG	Debug flag
ALL_EXC_TO_SENTRY	Route common exceptions to Sentry (flag present; Sentry wiring not found in settings — see note)
DB_NAME, DB_USER, DB_PASSWORD, DB_HOST	PostgreSQL connection (port hardcoded 5432)
AWS_REGION_NAME, AWS_REGION_NAME_2, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_BUCKET_NAME	AWS S3
CURRENCY_FAST_FOREX_API_KEY, CURRENCY_API_KEY	Currency APIs
STOCK_API_KEY	MarketStack
GROQ_API_KEY (and code reads GROQ_API)	Groq LLM
OPENAI_API_KEY (and code reads TOM_AI_PRIMARY_TOKEN)	OpenAI
AI_SUMMARY_ENABLED	AI summary feature flag
EMAIL_HOST_USER, EMAIL_HOST_PASSWORD	Office365 SMTP
CELERY_BROKER_URL, CELERY_RESULT_BACKEND	Redis (default localhost)
IN_DOCKER	Switches DB host in local/docker settings

⚠️ Env-var naming mismatch: .env.example documents GROQ_API_KEY and OPENAI_API_KEY, but the code reads GROQ_API (ai_insights/langchain_helper.py) and TOM_AI_PRIMARY_TOKEN (ai_module, csv_uploader mapping). Both the documented and the actually-read names must be set. (Fact — verify env on deploy.)

Config files

• settings/base.py + per-env modules; .env (from .env.example); docker/env/local/*.env; pytest.ini, conftest.py, .coveragerc, mypy.ini, .flake8, .isort.cfg, prospector_profile.yaml, .pre-commit-config.yaml; Dockerfile, docker-compose*.yml, appspec.yml + codedeploy/* (AWS CodeDeploy), procfile (Gunicorn).

Feature flags

• AI_SUMMARY_ENABLED (env). MAPPING_AI_PARENT_HEURISTIC_ONLY and many tunable weights/thresholds in the AI mapping service (env-driven). No general-purpose feature-flag framework.

Secrets

• All secrets come from environment variables (no committed secrets in settings). However docker-compose.yml contains hardcoded credentials for the Dozzle log viewer and DB env files — acceptable for local only. Do not reuse those in any shared environment.

Sentry note: ALL_EXC_TO_SENTRY exists in .env.example, but no Sentry SDK initialization is present in the settings modules examined. Unable to determine from the codebase whether Sentry is wired elsewhere (e.g. an untracked settings module).

---

13. Logging & Error Handling

Logging strategy

• Django LOGGING config (settings/base.py) defines a single console handler (StreamHandler) with a simple formatter and one named logger: csv_uploader.services.mapping_ai_service at INFO.
• Individual modules use logging.getLogger(__name__) (Celery tasks, AI services).
• Heavy use of print() across many modules (auth, AI, benchmarking, dashboards, CSV) instead of the logging framework — a consistent smell (§18). These print to stdout and are captured by container logging / Dozzle.

Exception handling

• Custom exception hierarchy (tom_exceptions/exceptions.py): TomBaseException(APIException) → TomException (400), TomValidationException (400), TomResourceNotFoundException (404), TomResourceAlreadyExistException (409), TomUnauthorizedException (403). Each carries an error (a code/message from tom_exceptions/errors.py) and an optional value interpolated into the message.
• Central handler custom_exception_handler (registered as DRF EXCEPTION_HANDLER) maps exceptions to the TomResponse envelope:

flowchart TB
    EX[Exception raised] --> H{custom_exception_handler}
    H -->|Throttled| R429[429 code=1001]
    H -->|TomBaseException| RT["TomResponse: exc.error.code + message"]
    H -->|ValidationError| RV["400 VALIDATION_ERROR, is_validation_error=true"]
    H -->|Http404 / DoesNotExist| R404[404 RESOURCE_NOT_FOUND]
    H -->|PermissionDenied / NotAuthenticated| R401[UNAUTHORIZED_ERROR]
    H -->|InvalidToken / TokenError| RTK[INVALID_TOKEN]
    H -->|KeyError| RK[400 field invalid]
    H -->|ValueError| RVE[400 VALIDATION_ERROR]
    H -->|anything else| R500["500 SOMETHING_WENT_WRONG (traceback printed)"]

Error responses

• Always the TomResponse envelope with success:false, a numeric code, a message, and (for validation) an error object + is_validation_error:true. Error codes/messages are catalogued in tom_exceptions/errors.py (TomError, TomValidationError, TomResourceNotFoundError, TomAuthorizedError, JobGradeError, …).

Monitoring

• django-prometheus exposes /metrics (request counts/latencies, DB query metrics via the prometheus DB backend).
• Container logs viewable via Dozzle (docker-compose.yml).
• Application-level error tracking (Sentry) — see §12 note.

---

14. Performance

Caching

• Django cache used by ai_module/ai_insights (analysis results + AI summaries; TTLs via CacheTimeout, e.g. 600s) and by AI-mapping embeddings (7-day cache with versioned invalidation key). The forgot_password throttle also uses the cache.
• Assumption: cache backend is Redis in production (Redis is present as broker), but the CACHES setting is not explicitly defined in the settings examined → defaults to Django's local-memory cache unless configured elsewhere. Unable to fully determine from the codebase — verify the production CACHES config.

Pagination

• DRF PageNumberPagination, PAGE_SIZE = 10 (global). response/tom_pagination.py::TomPagination wraps paginated results in the envelope and honors a ?page_size= override. Most list endpoints paginate; …/all variants deliberately return unpaginated data.

Database optimization

• bulk_create is used for child-row persistence in CSV ingestion, offer components, and mapping replace.
• select_for_update guards offer version-code generation.
• Weaknesses (the dominant performance issue):
• N+1 everywhere — nested SerializerMethodFields re-query per row; list views generally lack select_related/prefetch_related.
• LBT OR-combines querysets in nested per-grade/per-company loops then deep-accesses related objects without prefetch — the highest-risk hotspot.
• get_user_filters and get_target_company_ids_by_grade_filter computed repeatedly per request.
• Counter columns updated via read-modify-write instead of F() (race-prone).
• Some get() methods build a queryset then discard and re-query.

Lazy loading & batch processing

• Django querysets are lazy by default; the issue is not enough eager loading where serializers traverse relations (opposite problem). Batch processing appears in CSV ingestion (bulk_create) and LBT report building (numpy vectorized stats).

Possible bottlenecks

1. Synchronous external HTTP inside request/transaction — currency conversion (per-record in dashboard/offer math, no timeout/cache), MarketStack stock fetch (inside company create transaction), LLM calls in ai_insights/ai_module (mitigated only by caching).
2. LBT report N+1 + giant god-methods (build() ~590 lines, generate_csv_and_upload() ~440 lines) — CPU + query heavy; correctly moved to Celery.
3. In-memory pivots (job_grades.GradePointsView loads all grades+companies and loops in Python).
4. Zip decompression in ReUploadView (compute_payload_size) — unbounded (zip-bomb risk, §15).

---

15. Security

Area	Status in this codebase
Authentication	JWT (simplejwt), 1-day access / 5-day refresh, refresh blacklisting on logout, account lockout after 5 failed logins, login throttle (10/min/email), forgot-password throttle (3/2h).
Authorization	Role/permission classes + row-level role.metafield scoping. ⚠️ The _has_permission "method not in list → allow" behavior makes stacked permissions per-verb and easy to misconfigure. Several endpoints under-scope company access (see below).
Input validation	DRF serializers on most endpoints; custom pandas validation for CSV. Some views read request.data/query_params directly (KeyError/ValueError risk, handled by the exception handler).
SQL injection	Low risk — Django ORM parameterizes queries; no significant raw SQL.
XSS	API is JSON-first. Offer email/PDF render HTML via templates + token replacement — user-supplied fields (candidate names, email body) are interpolated; ensure template escaping (Django templates auto-escape by default). Offer logo upload validates magic bytes (PNG/JPEG, blocks SVG).
CSRF	CsrfViewMiddleware enabled; JWT (stateless) endpoints are not CSRF-bound. One view (UploadCompanyBenefitPlanPolicyManualView) is explicitly csrf_exempt.
Rate limiting	DRF throttling: anon 100/day, user 1000/day, login 10/min, forgot_password 3/min (plus the custom 3/2h). Throttle exceed → 429 with code 1001.
Sensitive data	Secrets via env vars. Passwords hashed by Django. Forgot-password responses do not reveal account existence. But hardcoded creds exist in docker-compose.yml (local Dozzle/DB) — local-only.

Concrete security findings (from code review — see §18 for the full list)

• Unauthenticated endpoints: ReUploadView (permission_classes = []) decompresses attacker-supplied (optionally AES) zips with no size/depth limit → zip-bomb / decompression DoS. CountryView and CurrencyView expose reference data with no auth. JeDocsUploadView has no permission classes (any authenticated user can upload to any JE by pk).
• Weak company scoping / horizontal access: LBT custom-filtered/ and function/, all custom_jobs sub-function endpoints, and several get_queryset-by-pk views (LivePayBenchMarkingDownload, offer/JE retrieve) fetch by pk without confirming the object belongs to the URL's company — potential cross-tenant data access.
• Prompt-injection surface: user-uploaded job-function names/descriptions are interpolated into LLM prompts. The mapping pipeline mitigates via structured output + ID allowlists; the free-text summary prompts in ai_insights/ai_module are more exposed.
• Silent failure of FX conversion returns 1.0, which can silently produce wrong compensation numbers.
• CORS_ORIGIN_ALLOW_ALL = True — all origins allowed. Acceptable only if the API is strictly token-authenticated and not cookie-authenticated for sensitive actions; review for production.

---

16. Dependency Map

Module dependency graph (internal)

flowchart TB
    subgraph Foundation
        UTILS[utils]
        RESP[response]
        EXC[tom_exceptions]
        BASE[tombackend.model / settings / celery]
        SWAG[swagger]
    end
    subgraph Identity
        AUTH[authentication]
        APPS[tom_applications]
    end
    subgraph Shared
        SVC[services<br/>services.py / offer_formula.py / email_service.py]
    end
    subgraph Setup
        CS[tom_company_setup]
        GRADE[tom_grade]
        JG[job_grades]
        CJ[custom_jobs]
    end
    subgraph Comp
        COMP[tom_compensation]
        OFFER[tom_offer_modeler]
    end
    subgraph Features
        JE[job_evaluations]
        LBT[benchmarking_tools]
        CSV[csv_uploader]
        AII[ai_insights]
        AIM[ai_module]
        DASH[dashboard]
        S3[s3_bucket]
    end

    AUTH --> APPS
    AUTH --> UTILS
    CS --> AUTH
    CS --> APPS
    CS --> SVC
    CS -. delete-guard .-> OFFER
    CS -. delete-guard .-> COMP
    COMP --> CS
    COMP --> SVC
    COMP --> GRADE
    COMP --> AUTH
    COMP -. delete-guard .-> JE
    COMP -. delete-guard .-> OFFER
    OFFER --> COMP
    OFFER --> CS
    OFFER --> SVC
    OFFER --> GRADE
    JE --> CS
    JE --> COMP
    JE --> JG
    JE --> GRADE
    LBT --> COMP
    LBT --> OFFER
    LBT --> CS
    LBT --> GRADE
    LBT --> SVC
    CSV --> COMP
    CSV --> CS
    AII --> OFFER
    AII --> AIM
    AIM --> OFFER
    AIM --> COMP
    DASH --> OFFER
    DASH --> COMP
    DASH --> CS
    JG --> CS
    JG --> GRADE
    CJ --> CS
    GRADE --> CS
    SVC --> OFFER
    SVC --> COMP
    ALL[All apps] --> RESP
    ALL --> EXC
    ALL --> UTILS

Key observations: - tom_company_setup is the coupling hub (imported by nearly everyone), yet it also imports back into tom_compensation and tom_offer_modeler (via delete-guards) → circular dependency risk between "setup" and downstream domains. These back-references are done with function-local imports in places to avoid import cycles. - services/services.py is a second hub — it imports tom_offer_modeler and tom_compensation while being imported by almost every app. - The AI apps form a small cluster (ai_insights → ai_module), sharing the AIInsight table and cache utilities.

---

17. Developer Guide

Where to add things (the mental model)

I want to…	Do it here
Add a new API endpoint	Add a GenericAPIView subclass in the app's views.py, wire it in the app's urls.py (path is auto-prefixed by tombackend/urls.py). Follow the existing Create/GetList/GetAll/Retrieve pattern.
Add request validation	In a DRF serializer (serializers.py) — validate() for cross-field rules, field-level validators, or an entry in validators.py. Keep view bodies thin.
Add business logic	Prefer a service — a function in services/services.py (if cross-app/reusable) or a class in the app's services/ package. Serializers may host create/update logic (existing convention), but new non-trivial logic belongs in a service.
Add a model	The app's models.py; inherit BaseModel; set an explicit db_table; follow the is_active="TRUE"/None soft-delete convention (or a Boolean if you must — but be consistent within the domain). For versioned reference data, follow the Version + Data + child-scoping triad. Then python manage.py makemigrations && migrate.
Add a permission	A new Permissions enum value (utils/enums.py), a Can… class in authentication/permissions.py, and add it to the relevant role grouping in utils/constants.py. Attach to views via permission_classes.
Add row-level scoping	Read/write ids in role.metafield and filter through services/services.py getters (use RoleMetafieldService).
Add a background job	A @app.task/@shared_task in the app's tasks.py; enqueue with .delay()/.apply_async(); it is auto-discovered. Update stage/status fields for progress.
Add an integration	Wrap the client in a service; read keys from settings (env); add timeouts/retries (many existing calls lack them).
Add an email	A helper in services/email_service.py that calls _send_email.
Add a file upload	Use s3_bucket patterns (boto3 / default_storage); validate type/size.

How to run & debug locally

```bash

1. Environment

cp .env.example .env # fill in DB, AWS, API keys; set env=local, DEBUG=true export env=local # settings/init.py requires this OS var

2. Dependencies

python -m venv venv && source venv/bin/activate # (Windows: .\venv\Scripts\activate) pip install -r requirements.txt

3. Database (PostgreSQL running locally per .env)

python manage.py migrate python manage.py createsuperuser

4. Run

python manage.py runserver 0.0.0.0:8080

In local settings, CELERY_ALWAYS_EAGER=True → tasks run inline (no worker needed).

For async behavior, run a worker: celery -A tombackend worker -l info

```

• Docker: docker-compose up (brings up tom_db Postgres, pgAdmin, Dozzle logs, tom_api). Set IN_DOCKER=1.
• API docs: /swagger/ (login-protected). Metrics: /metrics. Django admin: /admin/.
• Debugging: DEBUG=true for tracebacks; the exception handler prints tracebacks for unexpected errors; watch stdout (many modules print()). docker-compose.debug.yml exists for debugger attach.

Testing

bash pytest # all tests (pytest-django, factory_boy fixtures) pytest --cov=. # with coverage pytest path::TestClass::test # a single test mypy tombackend # type check prospector --profile prospector_profile.yaml # static analysis Test config: pytest.ini, conftest.py, .coveragerc. Coverage is currently light (~19 test files/dirs) — heaviest in services/tests/ and authentication/v2/tests/.

Common workflow

1. Branch from the default branch.
2. Add/modify model → makemigrations → review the migration → migrate.
3. Add serializer validation + service logic + view + URL.
4. Add/attach permission classes; consider row-level scoping.
5. Write tests; run pytest, mypy, flake8/prospector (pre-commit hooks configured).
6. Verify via /swagger/ or curl; confirm the TomResponse envelope shape.

---

18. Code Quality Review

Findings below are drawn directly from the source. Severity is the author's engineering judgment.

Correctness bugs (high priority)

#	Location	Issue
1	csv_uploader/views.py (CompanyJobGradeUploaderView.post, CompanySalaryRangeUploaderView.post)	upload_data(...) is called twice, duplicating validation, emails, and potentially writes.
2	csv_uploader/views.py (CompanyCashAllowanceUploaderView)	Checks if status is None against the imported DRF status module, not status_param — the required-param guard never fires.
3	custom_jobs/views.py (RetrieveCustomJobFunctionView.delete)	Method signature omits the company_id URL kwarg → TypeError at call time (delete is broken).
4	custom_jobs/views.py (RetrieveCustomJobFunctionView.get)	Uses many=True over a single instance returned by the getter → error/misbehavior.
5	job_evaluations/services/job_evaluation.py::submit	Sets submitted_by/submitted_at which do not exist on the model or migrations → silently dropped.
6	job_grades/views.py (DeleteJobGradeMappingView.delete)	Deleting one pk hard-deletes all companies' rows sharing that grade range.
7	job_grades/serializers.py (UpdateJobGradeSerializer)	Conflict validation guarded by if self.instance, which the view never sets → dead validation.
8	tom_compensation/views.py	CreateDefaultCompanyShortTermIncentiveVersionsView / …LongTermIncentiveVersionsView declare the wrong serializer (CompanyCashAllowanceVersionSerializer).
9	tom_compensation/serializers.py (CreateCompanyBenefitPlanView.post)	Possible UnboundLocalError on an unnamed IntegrityError.
10	benchmarking_tools/services/report_creation.py::build (aggregate title block)	References loop-leaked variables (combined_offer, etc.) holding the last grade's querysets; guarded elsewhere with if 'x' in locals() (a smell) → aggregate stats can use the wrong grade's data.
11	ai_insights/utils.py::store_insight_log	Computes an "existing" dedupe check but then unconditionally creates a new row; also writes to ai_module.AIInsight, not ai_insights.AIInsightModel.

Duplicate logic

• Compensation math implemented three times (services/offer_formula.py, tom_offer_modeler/email.py, comparison_service.py) — drift risk between offer list, email/PDF, and comparison.
• ~9 identical set-active PATCH bodies in tom_compensation/views.py.
• Grade↔country intersection validation duplicated ~4× in tom_compensation/serializers.py; get_stock_data copied 4×.
• CSV service classes (client_csv.py, ~10 classes) copy the same upload_data/_save_data/_send_error_email/download_csv shape.
• dashboard/service.py duplicates dashboard/utils.py CI computations.
• ai_insights repeats boilerplate that ai_module already factored into AIViewMixin.
• Near-duplicate views: JE list/sort/search (~90% identical); offer full-list vs light-list; two currency-conversion methods.

Tight coupling

• tom_compensation imports a private helper (_update_role_metafield_with_resource) from tom_company_setup.views.
• tom_company_setup ↔ tom_compensation/tom_offer_modeler circular dependency (delete-guards), papered over with function-local imports.
• services/services.py imports downstream apps while being imported by nearly all of them.
• ai_insights writes to ai_module's table.

Large classes / functions

• LivePayBenchmarkingReportBuilder.build() (~590 lines) and generate_csv_and_upload() (~440 lines).
• tom_compensation/views.py (3126), serializers.py (3506); tom_company_setup/views.py (2117); csv_uploader/services/client_csv.py (2896), mapping_ai_service.py (1248).
• je_rules.py — 575-line hardcoded matrix.

Missing abstractions

• No shared base for CSV services, for the set-active version flip, for the "cannot delete because referenced" guard (copy-pasted across taxonomy views), or for the AI-view flow (only ai_module has the mixin).
• Three coexisting grading concepts (tom_grade ranks, job_grades point ranges, role.metafield grade ids) with no unifying model.

Performance issues

• Pervasive N+1 (nested SerializerMethodFields, missing select_related/prefetch_related), worst in LBT.
• Synchronous external HTTP inside requests/transactions (FX, stock, LLM) without timeout/retry/caching.
• Read-modify-write counters instead of F().
• In-memory pivots (job_grades.GradePointsView).

Security concerns

• Unauthenticated ReUploadView (zip-bomb), CountryView/CurrencyView, JeDocsUploadView.
• Weak company scoping on several by-pk / [IsAuthenticated]-only endpoints (cross-tenant risk).
• The _has_permission per-method allow behavior (easy to misconfigure).
• Prompt-injection surface in AI prompts.
• CORS_ORIGIN_ALLOW_ALL = True.
• Silent 1.0 FX fallback → wrong money.

Miscellaneous smells

• print() used for logging across auth, AI, benchmarking, dashboard, CSV; left-in debug prints and [TIMING]/hardcoded test values (current_bonus_target = 9, acceptance_deadline = "July 15, 2023").
• Inconsistent soft-delete ("TRUE"/None string vs Boolean).
• Magic numbers (currency id 138/197, sub-function threshold ≥6).
• Dead code / vestigial fields (stock_tracking_ids ArrayField, IntegerListField, empty service.py, validators.validate_business_unit no-op, *Old enums, misspelled validtors.py).
• safe_float rounds to int, silently dropping cents in offer math.
• Hard deletes of STI/LTI child rows lose history despite the soft-delete convention elsewhere.
• Env-var name mismatches (GROQ_API vs GROQ_API_KEY; TOM_AI_PRIMARY_TOKEN vs OPENAI_API_KEY).
• simple_history installed but disabled (middleware commented out; no HistoricalRecords fields) → the offer app hand-rolls OfferVersionHistory instead.

---

19. Improvement Opportunities

Refactoring

1. Unify the compensation formula. Collapse offer_formula.py, email.py, and comparison_service.py into one authoritative module; use Decimal end-to-end and stop rounding to int in safe_float.
2. Extract shared bases: a VersionedDatasetService (create version / set-active / soft-delete), a CSVDatasetService base, a "referential delete-guard" helper, and adopt AIViewMixin in ai_insights.
3. Split god files/methods: decompose LivePayBenchmarkingReportBuilder, and break tom_compensation/tom_company_setup views and serializers into per-resource modules.
4. Introduce a thin repository/query layer (or at least Manager/QuerySet methods) so N+1-safe querysets (select_related/prefetch_related) are defined once and reused.
5. Standardize soft-delete on one mechanism (recommend a nullable deleted_at + a custom manager) and migrate the string-"TRUE"/Boolean split.
6. Move validation and write logic out of serializers into services for new code; keep serializers to (de)serialization + declarative validation.

Scalability

1. Kill N+1 systematically (add prefetching to all list serializers; the single biggest win, especially LBT).
2. Make external calls resilient: add timeouts + retries (tenacity is already a dependency) + caching for FX/stock; never return a silent 1.0 for FX — surface an error or use a cached rate.
3. Use F() expressions for the report/usage counters to remove races.
4. Push more work to Celery (currency prefetch, dashboard aggregation for large tenants) and add a proper CACHES (Redis) definition to settings.
5. Add DB indexes for the hot filter columns (company_id + is_active + version) if not already covered by the unique constraints.

Maintainability

1. Replace all print() with the logging framework; add structured logging and (re-)enable Sentry (ALL_EXC_TO_SENTRY implies it was intended).
2. Remove dead code (*Old enums, unused fields/services, empty modules) and fix misspellings.
3. Externalize je_rules.py into data (DB or config) so the scoring matrix can change without a deploy; guard matrix lookups with .get() + clear errors.
4. Fix the env-var name mismatches and document the full required env set.
5. Raise test coverage, especially around the compensation formula, JE scoring, LBT statistics, and permission edge cases.
6. Fix the correctness bugs in §18 (double-upload, broken custom-job delete, JE submit fields, over-broad job-grade delete, wrong serializers, LBT variable leakage).

Architecture

1. Break the setup↔compensation/offer circular dependency — move the shared reference-checks behind an interface or a dedicated catalog/domain app, so tom_company_setup no longer imports downstream models.
2. Consolidate the three grading concepts behind one grading domain model.
3. Formalize the row-level authorization (role.metafield scoping) into a documented, tested policy layer rather than ad-hoc id-list reads scattered through services.py.
4. Separate AI concerns — merge ai_insights and ai_module (or clearly split responsibilities) and stop cross-writing the AIInsight table; add a provider abstraction so OpenAI/Groq are swappable.
5. Introduce API versioning platform-wide (only auth is versioned today) before breaking changes are needed.
6. Add a feature-flag mechanism rather than scattered env booleans.

---

20. System Overview & Architecture

What the system is

A modular-monolith Django REST backend serving five frontend apps (SSO, TOM, JE, LBT, Dashboard) over one HTTP API, backed by PostgreSQL, with Celery workers for background jobs and S3 for files. (Frontend detail: Frontend Documentation; product view: Product Handbook.)

Runtime topology (production)

flowchart TB
    subgraph Clients["5 Frontend SPAs"]
        SSO[SSO] & TOM[TOM] & JE[JE] & LBT[LBT] & DASH[Dashboard]
    end
    LB[HTTPS / reverse proxy]
    subgraph Server["Application server (EC2, Ubuntu)"]
        SUP[Supervisor]
        GUN[Gunicorn<br/>WSGI, Django]
        CEL[Celery worker]
        SUP --> GUN
        SUP --> CEL
    end
    DB[(PostgreSQL)]
    RED[(Redis<br/>broker + cache)]
    S3[(AWS S3)]
    EXT[OpenAI · Groq · Currency ·<br/>Stock · Office365 SMTP]

    Clients --> LB --> GUN
    GUN --> DB
    GUN --> RED
    GUN --> S3
    GUN --> EXT
    RED <--> CEL
    CEL --> DB
    CEL --> S3
    CEL --> EXT
    GUN -. /metrics .-> PROM[Prometheus]

Architectural summary

Dimension	Choice
Style	Modular monolith (one Django project, ~20 apps)
Layering	View → Serializer → Service → ORM
Sync work	Gunicorn (WSGI)
Async work	Celery + Redis
Data	PostgreSQL (versioned reference data pattern)
Files	AWS S3 (django-storages)
Process mgmt	Supervisor
Auth	JWT (shared cookie SSO across frontends)

Detailed patterns are in §3 Architecture. This section is the one-glance overview; the diagrams are consolidated in §21.

---

21. System Diagrams

A consolidated index of the key diagrams in this document, plus a component map. (Individual diagrams live in their sections; this is the map.)

Component map

flowchart LR
    subgraph "API layer"
        URLS[urls.py] --> AUTHV[Auth views]
        URLS --> BIZ[Business app views]
    end
    subgraph "Cross-cutting"
        PERM[permissions.py]
        EXC[exception_handler]
        RESP[TomResponse]
        SVC[services/*]
    end
    subgraph "Domains"
        COMP[tom_compensation]
        OFFER[tom_offer_modeler]
        JE[job_evaluations]
        LBT[benchmarking_tools]
        AI[ai_insights / ai_module]
        CSV[csv_uploader]
        SETUP[tom_company_setup]
    end
    BIZ --> PERM
    BIZ --> SVC
    BIZ --> RESP
    BIZ -.errors.-> EXC
    SVC --> COMP & OFFER & JE & LBT & AI & CSV & SETUP

Diagram index

Diagram	Section
High-level architecture	§1
Middleware chain	§5
Request lifecycle (sequence)	§5
Entity relationships (ER)	§8
Versioning state machine	§8
Login flow (sequence)	§9
Permission model	§9
Background tasks	§10
Integrations	§11
Exception routing	§13
Dependency map	§16
Runtime topology	§20
CI/CD pipeline	§24

---

22. API Reference

This is the reference contract for the API. The full endpoint catalogue by module is in §7 API Layer; this section documents the conventions a client/integrator needs.

Base & versioning

• Base path: all endpoints are under /api/….
• Versioning: only auth is versioned — /api/auth/ (v1) and /api/v2/auth/ (v2, current). Everything else is unversioned.
• Company scoping: most business endpoints follow /api/company/<company_id>/….
• Interactive docs: /swagger/ (login-protected, drf-yasg).

Authentication

Aspect	Value
Scheme	Bearer JWT — Authorization: Bearer <access_token>
Obtain	POST /api/v2/auth/login/
Refresh	POST /api/auth/refresh-token/
Access token lifetime	1 day
Refresh token lifetime	5 days
Logout	POST /api/auth/logout/ (blacklists refresh)

Standard response envelope

Every response uses this shape (TomResponse.get_response):

json { "success": true, "is_validation_error": false, "message": "Human-readable message", "code": 0, "data": {}, "error": {}, "is_paginated": false, "pagination": { "count": 0, "next": null, "previous": null, "page_size": 10 } }

Pagination

Param	Meaning	Default
page	Page number	1
page_size	Items per page	10 (global PAGE_SIZE)

Paginated responses set is_paginated: true and fill pagination. List endpoints ending in /all are intentionally unpaginated.

Common query parameters (list endpoints)

Param	Meaning
search	Case-insensitive partial match across whitelisted fields
sort_by	Field to sort by (per-view whitelist)
order	asc | desc (default desc)
statuses	Filter (e.g. offers by status)

HTTP status & error codes

Situation	HTTP	code
Success	200/201	0
Validation error	400	VALIDATION_ERROR (with is_validation_error: true)
Invalid/expired token	400/401	INVALID_TOKEN (1002)
Unauthorized	403	UNAUTHORIZED_ERROR
Not found	404	RESOURCE_NOT_FOUND
Throttled	429	1001
Server error	500	SOMETHING_WENT_WRONG

Error codes/messages are catalogued in tom_exceptions/errors.py. Full handler behavior: §13.

Rate limits (throttling)

Scope	Limit
anon	100/day
user	1000/day
login	10/min (per email)
forgot_password	3/min (plus a custom 3 / 2h)

Content types

• Requests/responses: application/json (default).
• File upload (CSV, logos, docs): multipart/form-data.
• CSV download endpoints return text/csv (the frontend auto-downloads these).

Representative endpoints

A short, high-value slice (full catalogue in §7):

Method	Path	Purpose
POST	/api/v2/auth/login/	Login → tokens + claims
GET	/api/v2/auth/verify-user/?application=&company_id=	Confirm per-app access
GET	/api/company/<cid>/offers/list/	Paginated offers
POST	/api/company/<cid>/offer/	Start an offer
POST	/api/company/<cid>/offer/<pk>/offer-modeller/	Save compensation model
GET	/api/company/<cid>/job-grades/	List job grades
POST	/api/company/<cid>/job-grade/upload/	Upload job-grade CSV
POST	/api/company/<cid>/benchmarking-report/live-pay/	Trigger LBT report (async)
POST	/api/company/<cid>/job-evaluations/<pk>/evaluate/	Run job evaluation

---

23. Version Control

Note: the analyzed snapshot is not a git repository (git reports no repo at the analysis path), so branch/commit history could not be inspected directly. The following is inferred from CI/CD config files present in the repo (.github/workflows/, buildspec.yml, appspec.yml).

Branching (inferred from workflows)

Branch	Role (from CI triggers)
main	Production line — github_actions.yml (CI) and prod.yml (deploy) trigger on push to main
dev	Development — github_actions.yml CI also runs on dev

The docs-automation workflows added for this project use prod (backend) and cicd (frontend) branches per the team's repo layout — see the automation setup guide. The canonical product branch names should be confirmed with the team (Unable to fully determine from the codebase).

CI on every push / PR (github_actions.yml)

flowchart LR
    PR[Push / Pull request<br/>main or dev] --> SETUP[Set up Python 3.9]
    SETUP --> DEPS[pip install -r requirements.txt]
    DEPS --> TEST[pytest]
    TEST --> LINT[prospector<br/>static analysis]

Tooling that guards commits

Tool	Config	Purpose
pre-commit	.pre-commit-config.yaml	Runs hooks before commit
flake8	.flake8	Style/lint
isort	.isort.cfg	Import ordering
mypy	mypy.ini	Type checking
prospector	prospector_profile.yaml	Aggregated static analysis (runs in CI)
pytest	pytest.ini, conftest.py	Tests (runs in CI)
Dependabot	.github/workflows/dependabot.yml	Dependency updates

Commit / PR conventions

• CI runs on every pull request and on pushes to main/dev, so PRs must pass pytest + prospector.
• Specific commit-message or PR-template conventions: Unable to determine from the available code.

---

24. Deployment & Maintenance

Two deployment mechanisms exist in the repo

The codebase contains config for both of these (confirm with the team which is live):

1. Self-hosted GitHub Actions runner (.github/workflows/prod.yml) — deploys via docker-compose on push to main.
2. AWS CodeDeploy (appspec.yml + codedeploy/*.sh) — lifecycle-hook based deploy to EC2.

Path A — GitHub Actions self-hosted deploy (prod.yml)

flowchart TB
    PUSH[Push to main] --> RUN[Self-hosted runner<br/>labels: prod, backend]
    RUN --> ENV[Write .env from<br/>secrets.BACKEND_ENV]
    ENV --> VENV[Activate venv<br/>pip install -r requirements.txt]
    VENV --> MIG[python manage.py migrate]
    MIG --> DC[docker compose -f docker-compose.prod.yml<br/>down && up -d]
    DC --> SUP[supervisorctl restart gunicorn]

Path B — AWS CodeDeploy (appspec.yml + codedeploy/)

Deploys to /home/ubuntu/tombackend, running hook scripts in order:

flowchart TB
    D[CodeDeploy] --> BI[BeforeInstall<br/>before_install.sh]
    BI -->|stop gunicorn+celery,<br/>backup app keep last 3,<br/>ensure venv, apt deps| AI[AfterInstall<br/>after_install.sh]
    AI -->|pip install, migrate,<br/>collectstatic, start services| ST[ApplicationStart<br/>start_server.sh]
    ST --> SP[ApplicationStop<br/>stop_server.sh]

Hook	Script	Does
BeforeInstall	before_install.sh	Stops gunicorn/celery, backs up current app (keeps last 3), ensures venv, installs libpq-dev/build-essential
AfterInstall	after_install.sh	pip install, manage.py migrate, collectstatic, starts gunicorn + celery via Supervisor, verifies they're running
ApplicationStart	start_server.sh	Start marker
ApplicationStop	stop_server.sh	Stop marker

Build (buildspec.yml — AWS CodeBuild)

• Pulls the .env from Secrets Manager key prod/tom/backend.
• Produces the deployable artifact (all files).

Runtime processes

Process	Command	Managed by
Web (WSGI)	gunicorn tombackend.wsgi --timeout 15 --keep-alive 5 (procfile)	Supervisor
Worker	celery -A tombackend worker	Supervisor
Static	WhiteNoise (in-process)	—

⚠️ Note: the procfile references tomebackend.wsgi (typo — should be tombackend.wsgi). Verify the live process command.

Environment / secrets

• Settings module chosen by the env OS var (local/dev/staging/prod) — see §12.
• Production secrets: AWS Secrets Manager (prod/tom/backend) or the GH Actions BACKEND_ENV secret, written to .env at deploy.
• Full env-var list: §12.

Database migrations

• Applied automatically on deploy (python manage.py migrate --noinput in both paths).
• Create new migrations locally with makemigrations, review them, commit. Never edit applied migrations.

Maintenance runbook

Task	How
Deploy	Push to main (Path A) or trigger CodeDeploy (Path B).
Roll back	Path B keeps the last 3 app backups (*_backup_<timestamp>) — restore one and restart Supervisor.
Restart services	sudo supervisorctl restart gunicorn / celery.
Check health	sudo supervisorctl status; hit /metrics; check /swagger/.
View logs	Container logs (Dozzle in compose) / Supervisor logs / stdout.
Run a management command	Activate the venv, python manage.py <command> (e.g. resend_otp_emails).
Rotate a secret	Update Secrets Manager / GH secret, redeploy.
Scheduled jobs	None configured (no Celery beat) — see §10.

Monitoring

• Prometheus metrics at /metrics (django-prometheus).
• Application error tracking (Sentry): flag exists (ALL_EXC_TO_SENTRY) but SDK wiring Unable to determine from the available code — see §12.

---

Appendix — Assumptions & "Unable to determine" items

• Python runtime: 3.8 (Dockerfile) is treated as authoritative despite Pipfile/Readme divergence. (Assumption)
• Cache backend: CACHES is not defined in the settings modules read; production cache backend (likely Redis) is unable to be determined from the codebase.
• Sentry: ALL_EXC_TO_SENTRY env flag exists but no Sentry init found — unable to determine whether it is wired elsewhere.
• Celery Beat: no beat_schedule present — no scheduled jobs exist in the codebase (only an on-demand management command).
• AI mapping version set-active views (GetCompanyJFMappingVersionsView, SetActiveCompanyJFMappingVersionView) exist but are not routed in urls.py.
• Authoritative source among the three compensation-math implementations is not documented in code.

Document generated from static analysis of the repository at backend-prod/. Line-number references reflect the state of the code at analysis time and may drift as the code evolves.