Status of this project's website vs .com official site

[braver]

The official website has launched. It's forked from the website from this repo. This is good news, it means the plan worked! 🎉

It does raise the question of how to continue. Perhaps that needs a vote.

As I understand it, this is mostly about the website. The underlying crawling and data collection architecture of thecrawl is used as-is by the official website and channel. ie. SublimeHQ isn't also crawling everything again on their own, it's just the website itself that has been forked and the data from the crawler is replicated to their servers. I might be completely wrong though!

Edit: turns out I was in fact completely wrong! SublimeHQ maintains their own crawler infrastructure.

Personally I believe the community is best served by one source of truth and one official website, that we can all put energy towards and make it the best it can be. If the .com moves slowly because it's official that's fine, it's a very mature website already.

Alternatively both websites continue to exist and work in the .io maybe flows into .com at SublimeHQ's discretion.

But in my opinion having 2 websites muddies everything for no clear benefit other than the community gets to keep their project to play around with. I think fragmentation is the main pitfall of open source, it spreads and dilutes the community's energy. In this particular case I believe the .com website is the only one that's really worth contributing to as it will reach the most users. Even if that's a bit slower to move. Even if it requires approval from SublimeHQ to merge contributions.

In other words, I believe the goal has to be that .com replaces .io. One Package Control project with one website and source of truth (insert pop culture references to the Matrix, Highlander, Lord of the Rings, etc). But perhaps others have different opinions, so let's hear it from all the random cooks™️ 😄 @kaste @michaelblyons @TerminalFi @deathaxe @BenjaminSchaaf

---

See also #458 (comment) for more of my thoughts on the matter.

[BenjaminSchaaf]

Thanks for raising this discussion. I'm just going to put my stance out there and perhaps add some context.

We are running our own copy of the crawler. Technically it's also a fork of this repository, but the biggest thing I added was a deployment script, so that doesn't really count. Neither the crawler nor website use github's infrastructure.

Taking over project ownership is something sublimehq and I are ready and willing to do, but I'm not going to ask or demand such a thing. I fully understand being attached to a project you've put hard work into, so I don't want to pressure anyone into handing it over.

In the case where this project stays independent, I plan on keeping the website/crawler as close to this project as possible, upstreaming relevant changes as much as I can. If desired, some branding changes could be made to the .io website to make it more distinct, perhaps marking it as a staging area.

Once again, huge thanks to you all for making this project a reality. The official website would not exist without all your hard work.

[braver]

If desired, some branding changes could be made to the .io website to make it more distinct, perhaps marking it as a staging area.

Most of the current differences can be merged back into .io IMO, I don't see anything in there that's a problem.

While I'm not against the concept, on various levels I don't see a strong reason for there to be a "staging area" that is also a live website. To me that's just confusing. We've made the website extremely easy to run on your local machine and hack on it. Everyone can do that, try things, see what they look and feel like. A proposal for change can very easily be evaluated locally without having to be delivered to the public web.

However, I can see good reasons for there to be an air gap between the community project and the official website. It has a brand name on it after all. A good arrangement might be us maintaining the source project here and you regularly syncing up. But then I would still take the .io website offline and redirect traffic to .com. And put in some effort to keep the diff small so that there can be one issue tracker for the entire project. That clarifies the arrangement, and puts a little pressure on the .com to stay up to date. A little bit like how the packages project works where the packages don't automatically appear in ST, but everyone understands they eventually will after some action from HQ: there it's clear everyone's working on the same thing. We can then understand the fork is behind and HQ has final say on what happens when, but that arrangement can be explained and then how to contribute makes sense again.

I think the most important thing for contributors, and especially for myself, is the knowledge that you're contributing to the real thing, even if slightly indirectly. Not the community copy of it that nobody uses, but the real thing that everyone uses.

We are running our own copy of the crawler.

Similarly, if the crawler is not the real crawler, why would the community be interested in contributing to it? In keeping it up and running? The idea was always to become the thing, the effort to continue is only worth it if this is the thing. If we didn't believe we could completely replace packagecontrol.io we would never have started. Also, very practically, the crawler is a lot of work to build and improve and maintain, much more so than the website, and very few people know enough to do it. It doesn't make sense for there to be 2.

[michaelblyons]

Despite having put in the least work of the pinged people up above, I have Thoughts.

I'm glad that SHQ is here talking about things. It's an MIT project, so they're not legally obligated. They're just being considerate. Obviously, I have less invested in the control of the project, since I never had any of that to begin with. For Kaste and Braver, I'm sure there's some feelings there.

I loosely track with Braver's observations:

• Having one site is ostensibly desirable.¹
• People making contributions want the opportunity to have those incorporated, not siloed indefinitely into a toy site.

I do have some concerns about going to one site, though.

Operations visibility changes

Having some insight into the actual crawler operation on the .io site has been really cool, even though I don't have the keys to any of it. This lends credibility to things that were once opaque, like the "Needs Review" tagging. By contrast, the Status page is missing on the official site.

The opportunity to download the past year's package stats is also neat. If there are Business Reasons why stats and logs can't be public, I would be disappointed.

Contributing to hidden code

Whether the contribution point to the website is owned by SHQ or otherwise, if there is non-public code, it becomes hard to suggest changes to them (or even to know what they are).² Even in places where the changes themselves are obvious like the About page, where do I PR for Markdown that only Ben can see? If the production version becomes The Version with its source visible, maybe this is moot.

Braver already covered the "Who decides what gets accepted?" issue.

Shared infrastructure things

There are URLs various places that load from the .io site. In addition to the ones that Braver noted at sublimehq#1, the repackager³ still points to the CloudFlare site.

It sounds like SHQ plans to fully replace that stuff.

Footnotes

1. Having two sites has (mostly) worked for a year. Though amusingly, it turned out that PC.io is in some ways still a single point of failure (SPOF). ↩

2. I can't find it now, but I had some confusion over different implementations of PC's handling of SemVer prefix-stripping. I eventually found a years-earlier discovery of the same frustration by someone else. ↩

3. Uh-oh, the Repackager happily mangles the codeload zip for any repo into a *.sublime-package. That's kind of an open redirect problem, although when someone posts a link to a download, I guess the onus is on you to make sure that running it won't pwn your system. ↩

[deathaxe]

There's more possible confusion and fragmentation than advantages with maintaining separate repos and thus I'd prefer consistent, easy to follow, single source of truth infrastructure. Running two separate websites provides little value.

A github organization primarily organizes repos, logically.

If both crawler and website run on dedicated infrastructure not being coupled with github, does it mean any change made to the repo needs manual action to become active?

If yes, that's probably the air gap, required to prevent unwanted code changes to get online, already. Any unwanted change could be discussed and reverted at any time.

If that's not enough, protected production branches might be defined, which only sublimehq can push to. Community efforts could go into separate "develop" or "next" branches.

Working into sublimehq/thecrawl directly would then be very similar to sublimehq/Packages workflows, except the "two approvals required" rule might not hold (or maybe not needed?).

It also proved viable for Package Control Client and default channel, both of which are maintained by community in sublimehq/package_control and sublimehq/package_control_channel.

Having all building blocks located next to each other would be my fav:

• sublimehq/package_control_client
• sublimehq/package_control_channel
• sublimehq/package_control_crawler
• sublimehq/package_control_website (in case someone wants to separate thecrawl into front- and backend, instead of maintaining those parts in different branches of same repo)
• sublimehq/package_reviewer Kaste's package reviewer bot

It would also help to deploy consistent impersonated links to them instead of relying on personal repositories.

The github.com/packagecontrol org had originally been the place for vendored/wrapped dependencies. It hosting the libraries channel and thecrawl was decided due to no better alternatives. It's a pity it not even being the home for the currently primarily used package reviewer action.

---

About the stats page: The primary value is the list of failing or missing packages and the bot, which automatically proposes their removal.

[braver]

I would like to clarify my opinions as stated are just for me as a contributor. I feel proud of the project and maybe even a little ownership of the design of the website, but I have already moved away from it quite a bit. I’m just a sometimes contributor here, any time I have goes into reviewing package submissions. My contributions have been given with the full understanding of the MIT license 😄In other words whatever happens doesn’t affect me personally, I just hope for the clearest healthiest future for the entire project and those involved.

[michaelblyons]

Having all building blocks located next to each other… [list of repos]

Seconded. Also, perhaps the schema reviewer with a similarly consistent name.

[kaste]

Thanks for starting the discussion. I have been away, and the launch of the
.com site was a surprise to me as the project lead and code owner. There was
no prior agreement or even discussion to transfer this project, its governance,
or its canonical implementation to SublimeHQ.

The Discord announcement stated that the plan was to replace packagecontrol.io
entirely and “bring everything in-house,” and it tagged me while thanking me
for building the infrastructure. For clarity, I was not consulted about that
plan or involved in the decision. My mention should not be understood as
endorsement; the work being referenced was contributed to this community
project.

I agree with the narrow point that two nearly identical package websites are
not a desirable permanent outcome. However, this is not fundamentally a choice
between two websites, and I do not think a vote framed that way would be
useful.

Package Control consists of several independently maintained parts: the
registry and channel, the client installed by users, review and maintenance
tooling, and the crawler. The website is one consumer and presentation of the
crawler's artifacts. It is not the project's architectural or governance
centre.

In particular, thecrawl was deliberately designed as reusable, transparent
infrastructure. Its stated design goal is:

This project is built for an open world, not just “public source.”

The crawler runs publicly, its artifacts and logs are public, failures can be
reproduced locally, and its outputs can be used by tools other than this
website—including alternative Package Control clients. The gh-pages branch
exists because the website is only one possible view over those outputs.

Also note the auditable the-registry branch. This is our source of truth.

The website itself intentionally preserves the original packagecontrol.io
URL structure. It was designed that way so that the existing links throughout
the ecosystem would continue to work when the community implementation took
over the domain. The expected migration did not require establishing a second
site or breaking those links.

The new .com service currently runs a separate crawler and a private
production fork with undisclosed deployment differences. That is permitted by
the MIT licence, but it does not satisfy this project's transparency goals and
it does not make that deployment the canonical upstream. It also leaves
contributors unable to inspect production, understand operational failures, or
know whether work contributed here will reach users.

Nor is an informal promise to keep a downstream fork close to this repository
an adequate maintenance model. The contributions and synchronization so far
have required substantial maintainer review and correction. Proper changes to
this system require domain knowledge: DeathAxe maintains the client and knows
the schemas; Braver understands the wider ecosystem and maintains the channel;
and I know the channel, statistics, actual consumers and usage of the
artifacts, and the architecture of the crawler, which I wrote from scratch.
That knowledge is necessary to evaluate apparently small changes and avoid
regressions across the system. An unspecified downstream-sync process cannot
replace review by the people who understand these interacting parts.

The structure I had expected was different: the Package Control ecosystem
would remain under the community-owned packagecontrol organization;
packagecontrol.io and the associated Cloudflare account would be placed under
community stewardship; and SublimeHQ would participate in that organization
alongside the developers who maintain the actual components. Repositories for
the crawler, site, channel, and review tooling naturally belong together there
under the Package Control identity.

My preferred outcome remains one main package website at
packagecontrol.io, backed by the public crawler and artifacts. But again, the
actual URL is less important to me than thecrawl itself. The existing
crawler project page can then be reduced to the operational status,
statistics, documentation, and transparency interface appropriate to the
crawler itself. SublimeHQ may link or redirect its branded domain to that
service and is welcome to contribute through the same public process as
everyone else. I'm notoriously fast in handling issues and PRs.

I am not proposing that we transfer the project into sublimehq, nor that we
decide ownership through a vote. SublimeHQ has not historically maintained
Package Control's client, channel, crawler, or surrounding automation.
Launching a branded downstream service does not by itself transfer stewardship
of those projects.

Before deciding redirects or retiring any service, I would like SublimeHQ to
explain the decision to launch the separate implementation and discuss:

• transferring packagecontrol.io and its Cloudflare configuration to
  community stewardship;
• using the public crawler and its published artifacts rather than an opaque
  replacement;
• publishing any generally relevant production changes;
• retaining public status, logs, statistics, and reproducibility;
• establishing a clear contribution and deployment process; and
• participating in the existing packagecontrol organization with explicit
  and appropriate permissions.

That would give users one clear destination while preserving the openness,
reuse, compatibility, and community ownership on which Package Control has
been built.

[deathaxe]

I didn't want to be the first one, responding, but I find the last post quite unfair.

IMHO:

To be fair - without judging anything - critical parts (crawler, site) being hosted on sublimehq servers was - at least for me - the to be expected outcome. I am actually even under the impression it having been mentioned in former discussions. Even "packages.sublimetext.com" was announced and is IMHO the most logical choice with packagecontrol.io possibly being forwarded to it.

At least it was pretty clear packages.sublimetext.io or the infrastructure behind not being accepted as trustworthy enough for being the default choice. This was argued pretty clearly, before.

Trust and intent to not rely on US big tech infrastructure for such critical part are likely the most compelling reasons for this decision.

Package Control client is maintained directly in sublimehq/package_control repo by community member, with build artifacts being signed and copied to download.sublimetext.com/Package Control.sublime-package by sublimehq. From sublimehq perspective, I fully understand this step being a "manual review and approve" approach.

... SublimeHQ has not historically maintained Package Control's client, channel, crawler, or surrounding automation.

Despite that it has been requested multiple times by multiple users for SublimeHQ to take ownership of existing package infrastructure or create a new one - with the intent to ensure maintenance - instead of relying on external single point of failures, which at least the packagecontrol.io infrastructure turned out to be.

... Also note the auditable the-registry branch. This is our source of truth.

THE single source of truth IS https://github.com/sublimehq/package_control_channel

[braver]

critical parts (crawler, site) being hosted on sublimehq servers was - at least for me - the to be expected outcome.

I do also seem to remember this being the goal, but it's a bit hard to remember exactly how it was voiced on Discord. In any case I totally understand SublimeHQ wanting to own the entire "stack" and being free from GitHub's architecture. I've personally always hoped Package Control could be wholly owned by ST as that feels like the most stable situation: ST needs its package ecosystem and now they promise it continues to work as long as the editor continues to exist. To me that's all good news.

At the same time I do appreciate the open-ness of thecrawl, it's pretty great that all pieces of it are there for everyone to see. But it does run into the single-maintainer situation open source so often does. I don't want to be insensitive to the effort some of us so clearly put into this, but I personally feel better about it having a larger entity behind it. Even when it's fully open it's too much for a single person to be responsible for: open-source doesn't mean anyone else knows enough to meaningfully contribute. I understand I'm also not as deeply invested in the project itself, again my contributions here have been "gifted" and I hope people with more time do some good with it. I also only really contributed to the website, and that's been effectively "done" for ages now. It's relatively easy for me to walk away from that - as I've already done (and believe me, even that wasn't super easy).

For me this stands entirely separate from the channel. Which indeed I do maintain right now, but I'm not alone there and I still see that as an entirely voluntary and temporary position that doesn't really need me. I bring very little "domain knowledge" to that, of the people in this thread I probably know the least of how anything really works (I'm always really happy to see people contribute color schemes and syntaxes because those I can actually understand 😅), so lot's of people can easily step in and help (and they do!). I definitely don't "understand the wider ecosystem" any better than the people who taught me: the community is much broader than the 5 random cooks and Discord is full of people that know a ton more than I do. The reviews we do are also not a security layer, an approved PR does not constitute trust or even quality. The review process does not prevent hostile takeovers or malicious code getting inserted into a package. What I bring there is purely time and motivation cause I weirdly enjoy doing it. That's different from understanding how the Package Control client and crawler work, which are much more crucial and sensitive and would IMHO be better if they're not purely community projects (especially if in reality it's a community of one) and have some oversight or air gap.

[kaste]

Understood.

I don't want to be insensitive to the effort some of us so clearly put into this

Personally, I don't care. I'm also not attached to this project emotionally.
Believe it or not, it is just diligence I guess and restless note-taking.
It is: I started this shit, so I implement whatever they want.

... it's a community of one

That's interesting and a fear I have too, but almost the other way around.
sublimehq doesn't have much resources, human power; so my effort was helping
them out; let them focus on their software, and whatever their expert, domain
knowledge is.

The actual fear is rather that the development stalls and is killed, tbh.

too much for a single person

Not really, and I thought of it as normal internal dev tools work. You raise
your hand to do it when it comes up in a meeting, but it is not enough to build
a team. You commit, then do it for 1, 2, or ten years; usually you also succeed
on prior code. It is not only in OSS that you have some "single" responsibility.

anyone else knows enough to meaningfully contribute

I don't just know or understand the work of my colleagues. But I will if I
commit to do so. It is just romantic to assume you know the dirty corners in
your companies git repo.

--

So with that out of the way: to me, it is about closed vs open. I'm attached to
this idea. The idea, is simply put, with open code, open infra, and published
artifacts, people can stand on these shoulders.

sublimehq can claim the name, and run on different infra, but they didn't had
to close everything.

We actually have tools that read these artifacts -- it is not just the website.
Of yourse it is interplay between dev tools, the reviewer needs help from the
crawler. The 404 bot relies on data. Often, you have to change code on both
sides to accomplish a task.

Someone has to do that in a timely manner, or we fall back to prior state, remove
the 404 bot, the new reviewer and stare at needs_review tags.

10 weeks of commitment.

[deathaxe]

Without any insights I wonder about the following ...

To run the crawler/website on a normal web server, what is being needed?

Isn't it likely just a local clone of this repo and a set of scripts, which replace github workflows? Maybe a cron job to schedule crawling packages, produce channel.json and update static site?

I mean, everything is built to run locally, right?

Is that really such a critical part or rather some specific kind of glue code of minor value for the public?

---

About external tooling, wouldn't it be enough for registry.json and workspace.json to be published as downloadable assets, next to channel_v4.json?

---

The primary question/concern is just whether/how/how often community efforts end in production.

Ideally by just cloning them frequently, assuming agreement about implementations and features are found (for e.g. Popular packages and Stats site).

I frequently visit Stats, but primarily to check for new dead packages. Agree a bot should keep doing that, but maybe filing PRs about dead packages may be part of the crawler, just like generating final channel.json?

[michaelblyons]

but maybe filing PRs about dead packages may be part of the crawler

I thought this already existed. #353