Thank you Texium, will do!

Good Idea 4-Stroke. I think I can add Alibaba too!

AvBaeyer, it works like this: You enter the IUPAC Name or CAS number, and it'll find suppliers that sell that compound, and display them. When you click on the display you'll get redirected to their page for that compound. I think I'll use the Pubchem API to see other names of same chemical so it'll automatically search for them all.. But I am still thinking about the smartest way to do that

Right now I also added a feature where it'll tell you if they need end-user-statement for that chemical

I have compiled a list of potential suppliers in the US. Keep in mind that I have not purchased anything from these sites, and so may need some vetting before adding them to your software. I tried to exclude sites already listed on this site. Also, I have not verified that all of these, sell to residential addresses.

Some of these I think will be a good addition. There is one that specifically sells solvents and other chemicals relatively cheaply, and for amateur purposes. There is also another site I found that sells polymer precursors. I hope it helps!

Attachment: supply.docx (14kB)
This file has been downloaded 95 times

[Edited on 12/17/2024 by chempyre235]

Quote: Originally posted by chempyre235

I have compiled a list of potential suppliers in the US. Keep in mind that I have not purchased anything from these sites, and so may need some vetting before adding them to your software. I tried to exclude sites already listed on this site. Also, I have not verified that all of these, sell to residential addresses. Some of these I think will be a good addition. There is one that specifically sells solvents and other chemicals relatively cheaply, and for amateur purposes. There is also another site I found that sells polymer precursors. I hope it helps! [Edited on 12/17/2024 by chempyre235]

It was a bit more dififcult to do in javascript, so I decided to do it in python. When I am done, I can always just do it in JS afterwards too.
Anyway, this is how it's looking in a console app so far. I am still adding the shipping, quantity, notes, etc.

I still need some beta-testers, if anybody is interested? I am still not done for the beta-version yet, so it's going to take a while still.
Also, it is a bit more difficult for elements. For example if you search "sodium", the top results are sodium chloride, etc, but I'll get to fixing that.

[Edited on 9-2-2025 by Maui3]

Quote: Originally posted by Maui3

1: I have not checked the website for Carolina Chemical, but for some of the european ones there will be a little popup saying that it is not sold to private individuals above a certain concentration or amount. My python script could potentially check for that.

Quote: Originally posted by Maui3

2: This was very difficult to get to work, so the way it works right now is like this: It uses the search link, for example: "example.com/search=" + the chemName. Then it uses beautifulsoup to extract the HTML, where it finds the product and price from their classname. As you noted, this can be quite unreliable - so I am definitely also considering possibly doing a daily check for each of the websites in the sitemap.xml (since it'll take too long to do for each individual search), whereafter it'll find the product name and price - possibly by finding the first currency-symbol and then the closest chemName to the price-element. Of course chemical names differ, so it'll run through the Pubchem API for all of the synonyms.

Quote: Originally posted by Maui3

EDIT: I would loved to have used the suppliers APIs, I really would - but most of the suppliers never replied back to me after I told them I would like an API. There was one that replied back saying they'll send my message to their "tech-team", after which they could send me the API. It's been three months, they have no intentions on doing that.

Well, sir, since it would be a client side API (if its used on their website), just keep an eye on your networking tab of your browser.

Example of how to do it with LigmaAldrich

1) ProductSearch query:


gets a response with unique product keys:


2) Using the unique productKeys, request the Pricing and Availability endpoint:


Which will give you the price:


I think this would almost have to be done for some of them that populate the UI using RESTful calls from the client side. Otherwise, whats the best option? Launch a headless browser to track the client side requests? Way more tedious.

Quote: Originally posted by Maui3

3: That's kind of what I considered too (like I said in 2) - but so far it only checks for individual searches. There is not database, yet. I also thought that if I was to do this, I would do it as one of the last things, since making it extract the price and chemical name is the most important thing.

Quote: Originally posted by Maui3

4: Yes, of course! Besides, if the chemical suppliers websites whole system and URLs change, I would much prefer if it could be fixed by others than me. Also, I am just a amateur chemist and amateur programmer, it would be very ineffecient and ignorant to only let me edit it and understand how it works.

Quote: Originally posted by Maui3

Yes I am structuring it like that.

Quote: Originally posted by Maui3

Yes - I'll show you the code. It's structured quite poorly now - with variables that are.. well.. badly named.

Quote: Originally posted by Maui3

I'll fix it and send it to you in just a bit. Where (and how) would you prefer me sending it to you?

Here's an example of how to use Chemsavers API endpoints.

Sending the query for toluene:


And you get back a pretty beefy response with all the related products, and their prices:


But how do you do it programmatically?

Grab the curl request command for that specific request:


Throw that in your console, add some jq to make it pretty and get the data you want:


Do you need it in a different language? Python you said? Use CurlConverter.com for that:


The same process would work for AmBeed:

Obviously it might be better to search by CAS for Ambeed, or you could just look through the results to make sure they're what you expect. I searched for "toluene" and got a lot of results, none of which were actually toluene.

Elementalscientific has something that you can query, it returns the price as HTML, but that makes parsing it a heck of a lot easier:


Before you go through and do all the work on the search code for your tool, I would say go through to each chemical supplier and see if they have a public facing API when you search on their website. Create a list of which ones do have API's, which don't, and which ones are a hybrid (some have an API for searches, then you'd need to parse the HTML for the price - example: Carolina Chemical).

[Edited on 10-2-2025 by SuperOxide]

Thank you SuperOxide!

I am still a bit confused with exactly what to do, but I have been researching it a bit, and I am still a bit confused .. but it helped!

There is this european website I want to include: https://www.laboratoriumdiscounter.nl/en/
I am having a hard time finding their public facing API. Maybe they do not have one? What do we do with those?

Thank you for your help! I really appreciate it!

Quote: Originally posted by Maui3

Thank you SuperOxide! I am still a bit confused with exactly what to do, but I have been researching it a bit, and I am still a bit confused .. but it helped!

Quote: Originally posted by Maui3

There is this european website I want to include: https://www.laboratoriumdiscounter.nl/en/ I am having a hard time finding their public facing API. Maybe they do not have one? What do we do with those?

Quote: Originally posted by Maui3

Yes I get what you mean, their websites search function is not very nice for IUPAC names. I think we will do CAS-numbers instead, then.

Quote: Originally posted by Maui3

So for the rest of them, that don't have an API (the smaller ones), do we just do my initial approach with them? .. which was using BeautifulSoup to get the HTML, and use the classes to find the price and product?

Yes, unfortunately. And that's the tedious part.

Lets look at DudaDiesel as an example. Querying for products is very easy, just need to cURL their seach.php page with the query parameter. If I wanted to search for acetone, I would query the URL: https://www.dudadiesel.com/search.php?query=acetone.
That would return a plain HTML page, no fancy AJAX queries to get the prices - they're all on that page.
But then parsing the page to get the prices is a bit of a nightmare:

1. The prices themselves don't start with a currency sign.
   
2. There are input fields for quantities, which do contain the prices in hidden inputs, but not all items have those. The freight items don't, for example.
   
3. The HTML nodes with the prices don't have any unique identifiers or classes that you can directly search for, neither do their parent elements, or their parent elements, etc.
   

1. The only way I can think of to get the price values from this mess, is to look for div.item_spec HTML elements that contain strings "QTY" and/or "$Ea.", then grab the next sibling div.item_spec that has a float value in double quotes - and assume that's the price.
   
2. Alternatively, since all of the price elements seem to be formatted like <b>1+</b><br />8.48, you could look for any div.item_spec elements that have content that matches the RegExp pattern <b>([1-9]+)\+<\/b><br \/>([1-9]+\.[1-9]+), which would give you the quantity and the prices (example of pattern).
   
3. Regardless of which of the above two parsing options you choose, you would need to narrow down each one to the specific <table> or <tbody> node for each individual product listing.
   

As you can probably guess, the above searching can be pretty tedious. And it would be even worse if the prices aren't listed on the search results page, so you would need to parse the search results page for links to the products, then cURL each of those pages and run similar logic to get the prices.
This gives me a headache even thinking about it, lol.

While I have't used Pythons BeautifulSoup before, it does look like it would be able to take care of this.

Do you have a list of websites you want to include? Post them here. Maybe I can help with a quick analysis of each.

And do you know much OOP? I think the best way to handle this would be to create an inheritable class that has the main common methods that would be used for any of the merchant websites (simple cURL functions, JSON parser functions, HTML parser functions, etc), then each website you want to query would have its own class that inherits the base class, and is in its own file that would be included/imported and used to query for prices on that website.

I have an example of that type of setup in a recent project I was working on. I created a Moonraker plugin module called moonraker-contrast which needs to be able to parse gcode files from different slicers, so each slicer has its own module file, which contains a class for that specific slicer type (each of which inherits from a generic slicer class that has the common methods), and each slicer module is included in the main contrast.py file.
This means if I need to change the logic for one slicer (or in your case, chemical supplier website), I can just modify that one file. And if anyone wants to add a new supplier to the search feature, they just need to create a new module/class that inherits the base class, and put it in the right directory.

But, first things first. Do you use Git (revision control)? If not, start using it. Sign up on Github.com (free), create a repository for this project, then install git on your local machine and commit some code to it. Start checking in your changes so you don't lose any of your files or anything.
- Install git
- Create a repo and add some code to it

Sorry if this is a lot of information for you to take in. I think this is an excellent learning opportunity for you though, and the project as a whole sounds like an excellent idea.

[Edited on 11-2-2025 by SuperOxide]

[Edited on 11-2-2025 by SuperOxide]

Quote: Originally posted by Maui3

This sounds like a great idea! I will start working on all of this (setting up github, coding, listing website, etc.)! For the list of websites, I can list the european ones, as I am from Europe. I do not know enough about where to buy in North America (or other places) to make a list for that. But this page could be helpful for that: https://www.sciencemadness.org/smwiki/index.php/Lab_supplier... For the european ones, this is what I've got: Laboratoriumdiscounter Pretty much everything (at a quite cheap price too) an amateur chemist needs for most projects. S3 Chemicals A lot of standard and also a few somewhat obscure reagents at really good prices. Onyxmet Has a lot of elements and obscure reagents that are otherwise hard to get for amatuers like NaBH4, etc. Not a lot of chemicals though. Laborladen Does really not have a lot of chemicals, but very cheap metals. Warchem Has a lot of reagents, which are also available in smaller quantities. Is also really cheap. Though they only sell to poland. Possibly Lerochem Has really cheap common chemicals. .. but I need to hear some other reviews for the place first. Hadron Scientific Has really cheap common chemicals, and a few obscure ones. There are more, but I feel this is the best to start with anyway.

Quote: Originally posted by Maui3

For the list of websites, I can list the european ones, as I am from Europe. I do not know enough about where to buy in North America (or other places) to make a list for that.

Quote: Originally posted by bnull

Sites may change from time to time, the code changing along with them. I recommend you inspect them occasionally to make sure your code is not outputting something else. Color codes, for example, or gibberish.

You can visit this in your browser and see that there's several sitemap files, one of which is the cbs-product-sitemap.xml which looks pretty useful. You can visit this in your browser as well (viewing it in your developer tools Elements pane makes it a lot easier to read, btw). But this only seems to contain the links, update frequency and priority for each URL. That might be helpful if you could find the last time it was updated, but I don't see where that is.

But the tricky part is that some websites don't want you to access them through anything but an actual browser. If you curl the sitemap file in the command line you'll get back a default response saying you need to enable javascript to view the page. This is usually checked for on the server side by looking at your User Agent value and making sure its a valid browser. This can typically be circumvented by just viewing the request in your browsers Network tab, right clicking on it and selecting "Copy as cURL". This should copy every header, coolie and parameter that was used to make the request from your browser....

Only, it doesn't seem to work for Carolina:

If you look at the responses, youll see it sets some data in a dd variable, then loads a javascript file which seems to be doing the cloaking. I see its doing something with an iframe.. But I would think that if there was an iframe, I would be able to see it.

Point is - Some websites are going to have roadblocks like this that will slow things down quite a bit. So expect to have some unexpected fun along the way. lol

P.S. Carolina left developer keys in the open (just look for "key"), lol. Hopefully those aren't actual access keys.

P.S.S. Im noticing for Carolina.com, if I open a new incognito window and go to their website, it first will prompt for a captcha. I'm not sure how this could be circumvented from the command line. I hope not too many sites are like this.

[Edited on 12-2-2025 by SuperOxide]

Thank you all, this sounds good!

SuperOxide, It looks very nice with S3 Chemicals and Laboratoriumdiscounter, but for Onyxmet and the beautifulsoup, it says this?

I presume this isn't great, lol.

Quote: Originally posted by Maui3

Thank you all, this sounds good! SuperOxide, It looks very nice with S3 Chemicals and Laboratoriumdiscounter, but for Onyxmet and the beautifulsoup, it says this? I presume this isn't great, lol.

Quote: Originally posted by bnull

Use HTTrack to save a copy of the site. You don't need the pictures, only the code, and HTTrack will preserve the structure.

Quote: Originally posted by Maui3

Great Superoxide! I added the code you wrote for Onyxmet to mine, and now it works! Here is a video: https://i.imgur.com/7ITFNZl.mp4

Quote: Originally posted by Maui3

Great Superoxide! I will send it to you soon, I just need to add more comments - which I am not very good at..

Quote: Originally posted by Maui3

Also, for the laboratoriumdiscounter and S3 chemicals, I only use beautifulsoup to get the HTML.. We will also need to add a "quantity" for some of them, since they might not be specified in the title. For laboratoriumdiscounter, sometimes they do, other times they dont. They really make it easy for us, lol..

Quote: Originally posted by Maui3

I don't know if it is possible, but can I add you as a co-owner? If that is a thing, lol.

Quote: Originally posted by Maui3

I have added you now as a collaborator. Also, it was difficult to upload the code - the folders didn't upload.. I need to fix that..

Quote: Originally posted by SuperOxide

But, first things first. Do you use Git (revision control)? If not, start using it. Sign up on Github.com (free), create a repository for this project, then install git on your local machine and commit some code to it. Start checking in your changes so you don't lose any of your files or anything. - Install git - Create a repo and add some code to it

Room for another?

Quote: Originally posted by imidazole

I'm an okay programmer, and I sent this post from Linux, are you looking for more hands?