Web scraping in 10 mins

• Newman

  250 6,724 7.5 JavaScript

  Newman is a command-line collection runner for Postman

  

Well, there is this website that I have been trying to scrape for a few days now. I had tried everything from scrapy splash on docker to almost giving up because I read somewhere that it was JavaScript rendered. Since the source code from the inspect part of the developer tools was different from the source code from the view-source:https//... on the same developer tools.How could this be possible? Then I kept searching on internet and found this concept; where you can mimic web-browsers requests from a server using an API program,and it worked magically. Some of the API programs are postman and insomnia. I prefer using insomnia for this particular case , feel free to use any other API program of your choice.

• insomnia

  225 33,244 9.7 JavaScript

  The open-source, cross-platform API client for GraphQL, REST, WebSockets, SSE and gRPC. With Cloud, Local and Git storage.

  

Well, there is this website that I have been trying to scrape for a few days now. I had tried everything from scrapy splash on docker to almost giving up because I read somewhere that it was JavaScript rendered. Since the source code from the inspect part of the developer tools was different from the source code from the view-source:https//... on the same developer tools.How could this be possible? Then I kept searching on internet and found this concept; where you can mimic web-browsers requests from a server using an API program,and it worked magically. Some of the API programs are postman and insomnia. I prefer using insomnia for this particular case , feel free to use any other API program of your choice.

• SurveyJS

  surveyjs.io featured

  Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

  

Suggest a related project