Earlier this week, we added a couple features named “Entity Enrichment” and “Entity Aliasing”. These features allowed users to get a better picture of an entity through techniques such as DNS and clever ways of comparing entities.

Now, to make this even easier, we’ve added a way to show these correlated entities as a single unit. This makes seeing an entity with many different names easier.

Announcing the “Meta-Entity” view:

Now you can see there are 107 unique “meta-entities” vs the standard view with each entity shown as its own row (202). Previously, you’d need to do this correlation yourself on the Entities page, and it was difficult to tell if you’d already looked into an entity because many lines could represent the same entity, just a different view.

Comments & feedback welcome!

 

 

Announcing two new features called “entity enrichment” and “entity aliasing”.

Entity enrichment allows us to get a more complete picture of an entity. It is a process that happens automatically for certain entities upon their creation. Today IpAddress, DnsRecord, and Uri are supported. For each of these entity types, one or several enrichment tasks will be run as soon as the entity is created, allowing us to discover additional facts and alternate names (“aliases”) for the entity.

How it works: Upon creation, an enrichment task (lib/tasks/enrich/*) will be scheduled and run. In the case of an IpAddress, the name will be resolved to A, CNAME, or PTR records, DnsRecord entities created,  and finally “aliased” to the IpAddress.

Let’s show a quick demo.

First, create a new project and select the “Create Entity” task with a DnsRecord entity. In this case, we’ll use the DnsRecord “intrigue.io” with no recursive depth:

Hit “Run Task” and you’ll see that it kicks off the task, creating the entity:

Now browse to the entities page. Notice that there are now 3 total entities, one DnsRecord, and two IpAddress entities. Note also that the IpAddress entities are both are aliased to the “intrigue.io” DnsRecord. In this way, we can quickly find load balancers and other interesting DNS configurations.

Now, let’s try with a larger iteration strategy (3):

Give it a few moments, and now, on the Entities view, filtering for IpAddress only, we can see the correlation of IpAddress to DnsRecord:

This is also a good way to find DNS entries that are no longer active or resolving to an IP, but this is left as an exercise for the reader.

 

So you’ve gotten an instance of intrigue-core up and running using the AMI or Docker guide, but what now!? Give scans a try. Here’s now.

Create a new project, let’s run this one on Mastercard (They run a public bounty on Bugcrowd):

Now, run a “Create Entity” task to create a DnsRecord with the name “mastercard.com”.

This time, however, let’s set our recursive depth to 3. This will tell the system to run all viable tasks when a new entity is created, recursing until we reach our maximum depth:

Hit “Run Task” and you’ll see that our entity was successfully created:

Now, let’s browse to the “Results” tab and get an overview of the “Autoscheduled Tasks” that have been kicked off automatically:

Wow, 83 tasks in just a few seconds! Core is FAST, thanks to Sidekiq and Sequel. Now we can browse over to the “Graph” tab, and get an overview of the entities (nodes) and the tasks (edges) that created them.

Note that the graph is generated every time you load the page, so you will need to refresh a couple times to get the graph to show. You can zoom in and out to get details on the nodes:

Browsing over to the “Dossier”, you can see that there’s some fingerprinting happening on the webservers, based on the page contents. Note that there’s nothing invasive happening here, this is simply just doing page grabs and analyzing the results:

One neat feature is that core actually parses web content – including PDFs and other file formats to pull out metadata. More to come on this!

All this in just a few minutes: 

 

To get started with intrigue-core using Docker, you’ll need to install Docker on your machine.

Next, pull down the intrigue-core repository to your local machine with a git clone:

$ git clone https://github.com/intrigueio/intrigue-core
$ cd intrigue-core
$ docker build .
$ docker run -i -t -p 7777:7777 [image id]

This will start postgres, redis and the intrigue-core service, giving you output that looks like the following (shortened for brevity):

Starting PostgreSQL 9.6 database server                                                                                                                                                           [ OK ] 
Starting redis-server: redis-server.
Starting intrigue-core processes
[+] Setup initiated!
[+] Generating system password: hwphqlymmpfrqurv
[+] Copying puma config....
[ ] File already exists, skipping: /core/config/puma.rb

* Listening on tcp://0.0.0.0:7777
Use Ctrl-C to stop

As it starts up, you can see that it generates a unique password. You can  now log in with the username intrigue and the password above at http://localhost:7777 on your host machine!

• URI: http://localhost:7777
• Username: intrigue
• Password: [generated on startup, see above]

Now, to kick the tires, create a project:

Now that you have a new project, let’s run a single task:

This will give us lots of interesting things to look into:

And we can click on any of these and “iterate” with a new task. Let’s iterate on that first DnsRecord, sip.microsoft.com. Click on it and you’ll see all of the entity’s details, as well as the task runner below, where we can select from all tasks that can run on a DnsRecord entity.

In this case, we select “nmap scan” and hit “Run Task”, allowing us to iterate further!

Keep going, and see what you can discover.

Note that many tasks require an API key – which you configure in the “Configure” tab. Each listing has a handy link to the configuration, making it easy to provision an API key:

 

Don’t forget to check out the Dossier and Graph views to show you a listing of all entities and a graph of all entities respectively:

Oh, and by clicking on a node, you can get a bit more info in the HUD!

That’s it for now. Have fun! Please jump in the Gitter channel if you have troubles or want to learn more.

UPDATE: The latest test image can be found by searching ‘intrigue-core-edge’ in Community AMIs. It is currently only available in the Northern Virginia region on EC2.

I’ve made an EC2 instance available for testing if you’d like a simple way to try it out. Here’s a simple demo of how to get started.

Quick demo of the updated scan runner which is now dynamically populated, allowing you to quickly add new scan types, and run them with minimum manual effort.

I’m going to try out doing demo videos for new features. Here’s a demo of the new “Search OpenCorporates” module. It hits the OpenCorporates API and pulls back results. We could do a better job of parsing out useful information, but it’s a start.

Thanks to everyone who came out!

Here are the slides: slides

To view & download the code, visit the Github page.

For those interested in the zone transfer data, you can find a csv summary of results here.

 

 

Great reading from our friends at the CIA:

Not only are open sources increasingly accessible, ubiquitous, and valuable, but they can shine in particular against the hardest of hard targets. OSINT is at times the “INT” of first resort, last resort, and every resort in between.

https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol48no3/article05.html