Our job-index crawler

What it is

• A crawler named MyInterviewerJobIndex, version 1.0. It runs on a schedule, a few times a day, and identifies itself in every request with the User-Agent above, which points to this page.
• It reads the public job-board feeds that employers' applicant-tracking systems publish for careers pages. It does not read listing sites, social networks or search results.

What it fetches

Only the public feed or API of an employer's job board, on these vendors' hosts:

• Greenhouse: boards-api.greenhouse.io
• Lever: api.lever.co
• Ashby: api.ashbyhq.com
• Workable: apply.workable.com
• Teamtailor: careers sites on teamtailor.com, one subdomain per employer
• Traffit: careers sites on traffit.com, one subdomain per employer
• It never signs in, sends no cookies, does not try to get past a CAPTCHA or any other check, and does not follow redirects. It makes GET requests only.

How often and how gently

• Before it calls a host, it reads that host's robots.txt. It obeys Disallow rules for MyInterviewerJobIndex or for all robots, for the exact address it would call, and it obeys Crawl-delay (it waits up to a minute between calls).
• If robots.txt cannot be read because of a server error, it skips that host for the run. If there is no robots.txt (a 404), it carries on.
• It sends one request at a time to a host and waits between requests. When a host answers 429 or 503 it waits, as long as Retry-After says, tries only a few more times, and then leaves that host alone for the rest of the run.
• It asks for a feed again only when the feed may have changed (If-None-Match and If-Modified-Since), and refuses answers above a size limit.
• It reads the Content-Signal lines some sites publish in robots.txt and logs them. It does not train any AI model on what it reads.

What we store

• For each posting: the title, the location as written, a plain-text version of the description, the employer's own link, and the dates the employer gave. We add the employer's name from our own list.
• Nothing about any person who uses My Interviewer. Searches are made on our own copy, and nothing a person types is sent to an employer or to a vendor.
• Every result links to the employer's own page. A posting the employer removes is hidden at once after our next successful read and deleted a week later.
• We keep a log of our requests (host, address without any query, answer code, time taken, size). It holds no user data and is deleted after 90 days.
• Nothing we store is used to train AI models, ours or anyone else's. When a person chooses to practise for a posting, its text is used to write questions for that person only.

Ask us to stop or to remove postings

• Add a rule for MyInterviewerJobIndex (or for all robots) to your robots.txt. We read it before each run and keep it for up to 24 hours.
• Or write to us at the address below with the employer or site and what you want removed. We reply within 2 business days. Once we have checked that the request comes from the site or the employer, we remove the postings within 24 hours.
• We can stop reading one employer without waiting for a new release of our software.

Links a person pastes

• When a user pastes a job link into My Interviewer, our server opens that one page once, to read the job text for them. It does nothing else on your site: it does not follow links and does not come back.
• That fetch checks the site's robots.txt first. If the site does not allow it, we tell the person to paste the posting text instead. The same goes for an employer or site that has asked us to stop: we do not fetch its pages for a pasted link either.
• Our own request for that page is sent with an ordinary browser User-Agent, not the crawler's, because many careers pages turn away unfamiliar ones. Only the robots.txt check is sent with the crawler's User-Agent above, and it is judged by the same rules. If the link redirects, we check every address along the way (robots.txt, the list of sites we do not fetch, sign-in pages), and we follow only a short chain of redirects and stop at a loop.
• If our own request does not return the posting, for example because the site turns it away, we may pass that one link to a third-party reader, Jina AI (Jina Reader), which fetches the page itself and sends back plain text. We use it only for a link a person pasted, never for the crawler. We do not use it when the site's robots.txt says no, when the site is on our list of sites we do not fetch, or when the page is a sign-in page. Jina fetches with its own User-Agent and follows its own rules, which we do not control. Jina is named among the companies that process data for us in our Privacy Policy.

The preparation assistant

• A person can ask the optional preparation assistant to read a company page they typed into their own plan, to write a short, sourced brief for them. It opens only the addresses the person typed, once, never follows a link it finds on a page, and sends the crawler's User-Agent above, so one name in your logs and one robots.txt group, MyInterviewerJobIndex, covers it.
• Before it opens a page it reads that site's robots.txt and is judged by the same rules as the crawler. If the site does not allow it, the brief says the page was not read.
• In short: the crawler and the assistant's page reads send the User-Agent above; for a pasted job link only the robots.txt check does (see the section on pasted links). All three are judged under MyInterviewerJobIndex, or under all robots when you have no group for it.