Redditology / Source / System 01 of 03
2022Recorded work
Creator · experimental side project
Source relationships and content selection
Choose suitable discussions, keep comments with the posts they answer and exclude text that will not work in the video.
- Noah’s contribution
- Noah specified the source structure, configured relationships and filters, and revised selection rules.
- Historical stage
- Historical manual handoff, later scraper/callback workflow and evolving filters.
- Part of the business
- Redditology overview ↗
- Airtable
- Google Sheets
- Apify
- Make
Inside this system
The workflow
The handoffs
inside the work.
Follow the main path and its branches. Select a step for context; the account below explains what was built, tested or planned.
- Prepared Airtable post queue
Retain post identity, source URL and suitability/readiness state. - Operator selects suitable post
Keep low-quality material and excessively long titles outside the production queue. - Apify comment-collection run
Start collection for the selected post and save its run ID on the post. - Scrape-completion callback
Use the returned run identifier to recover the initiating post. - Post, comment and reply records
Preserve comment IDs, source-post links and immediate-parent relationships. - Suitability filters
Apply text, URL, deleted/removed-content and layout-length exclusions. - Excluded source material
Keep unsuitable content out of the selected production material. - Selected material for generation
Pass cleaned text and source/parent identity into asset preparation. - Inspect missing discussion content
Review reply losses and revise experimental ratio filters where necessary.
Key records
The information
that carries forward.
| Record | Information | Purpose |
|---|---|---|
| Post | Post ID, title, URL, creation details and selection state | Identify the content unit and its place in the queue. |
| Scrape run | Run ID stored against the post | Match a completion callback to the initiating post. |
| Comment or reply | Comment ID, parent ID, text and score | Preserve nested discussion relationships. |
| Selection state | Suitability and readiness labels | Separate rejected material from work ready to prepare. |
Inside the system / 01
A consistent source record before automation
An early operating handoff asked a teammate to collect posts and comments into a spreadsheet. The requested information included text, author, date and score, with the parent post ID retained beside each comment. Newline cleanup and consistent formatting were part of the task. This version was a human collection process.
Comments had to stay connected to their original post, and replies to the comments they answered. Those relationships let later production steps preserve the discussion’s order.
Inside the system / 02
A callback needs a path back to the post
Later walkthroughs describe a prepared post list in Airtable and an Apify run that collected comments when a post entered production. The run ID was saved against the post. When the completion callback arrived, that identifier allowed the next scenario to recover the correct post and associate its returned comments.
Collecting comments at that point limited the amount of source data acquired before it was needed. Post IDs, run IDs and parent IDs served different roles: selecting the content unit, matching the asynchronous run and preserving the discussion hierarchy.
Inside the system / 03
Filters affect the story, as well as the workload
The historical rules considered deleted or removed text, URLs, length and formatting. Links could produce awkward narration, while very long text could exceed the visual layout. An operator also marked posts unsuitable or their titles too long before moving useful material forward.
Upvote ratios were an evolving experiment rather than a settled quality measure. One repair session found that reply filters had removed almost everything after the parent comments passed. Noah removed or adjusted those ratios to recover more replies while retaining other exclusions. Missing scraper replies were a separate concern.
These revisions show why selection logic needed inspection against the resulting discussion. A filter could execute successfully and still remove the material needed for a coherent output. The recordings establish changes to those rules, rather than an objectively validated scoring model.
Within Redditology