Kristina Hoeppner 00:05

Welcome to 'Create. Share. Engage.' This is the podcast about portfolios for learning and more for educators, learning designers, and managers are keen on integrating portfolios with their education and professional development practices. 'Create. Share. Engage.' is brought to you by the Mahara team at Catalyst IT. My name is Kristina Hoeppner. 

Kristina Hoeppner 00:26

Today my guests are Professor Dr Paul Libbrecht and Professor Dr Wolfgang Müller from PH Weingarten, University of Education Weingarten in Germany. They are members of the AISOP project that looks at integrating artificial intelligence into portfolios that are based on Mahara. Welcome, Paul and Wolfgang.

Paul Libbrecht 00:47

Hello. 

Wolfgang Müller 00:48

Hello. 

Kristina Hoeppner 00:49

Can you please tell us a little bit about yourselves? What do you do? Let's start with Wolfgang, please.

Wolfgang Müller 00:55

I'm a Professor for Media Education as the University of Education Weingarten and my background is in fact computer science. I'm teaching here quite a bit in our study programme and media education and management, which has parts in media education, business, and also communication. Since some 15 years I've been applying ePortfolios in this context of our study programme. Paul.

Paul Libbrecht 01:16

I'm a Professor of Computer Science and Data Science at the IU (International University of Applied Science), and I'm a researcher in the AISOP project at the PH Weingarten, at the University of Education. I've been applying portfolios at the University of Education and at the IU with very different perspectives and evaluation methods. And my background is in mathematics. Within the AISOP project, I do a lot of things close to development. I'm also a developer of learning software.

Kristina Hoeppner 01:41

Thank you so much. So computer scientists, mathematicians, how do you get interested in portfolios? 

Paul Libbrecht 01:47

There is this feeling and especially when you are a computer scientist that the computer's science remains extremely abstract as long as you don't programme it. Applying this to portfolio is pretty easy. When a student that you have presented a course to needs to show his or her knowledge, then it's time to realise something, and you can request a simple presentation. But what's interesting is to request how the simple presentation has been coming. How this learning that has been provided is appearing. And I think this is a natural introduction to portfolios. We all want to see that learning is an authentic process that has happened. Generally, I'm really bothered when I see things that are just presenting back some knowledge that was learned before.

Wolfgang Müller 02:38

I think Paul's said probably the most important aspects already. I would like to add that yes, for me as a computer scientist, this was also a starting point thinking around the in terms of projects and self directed learning. And I always had this feeling that written exams or oral exams, it's difficult to grasp really the competencies of students and they don't motivate students. We were looking here from the very beginning to other forms of assessment, but we don't restrict ourselves on genuine computer science topics, but we really apply ePortfolios on a broader scale.

Kristina Hoeppner 03:07

In which study programmes at the University of Education do you use portfolios?

Wolfgang Müller 03:13

For the moment, the most important one is the one I mentioned before where we are teaching, which is one on Media Education and Management. We also use it in our teacher education programmes on computer science education for computer science teachers. And I think what we're planning right now also in the context of a larger project is to bring ePortfolios also as a mandatory form of assessment and of learning to the whole of our teacher education programme in this context of media education.

Kristina Hoeppner 03:41

Is that quite a new approach for the media education programme that you're in? 

Paul Libbrecht 03:47

For computer science that I have in my other university, I must say when I was requested to do portfolio, I knew what I was doing because I had experience from the University of Education. But honestly, no other colleague had experience in there. And it's it's pretty rare that people request this approach in a systematic fashion. You see a lot of project and project reporting, but the learning aspects, the design aspects, the conception aspect, the conception process aspect is something which I don't see often done. Every day coming back again and again, you know, lifelong learning, the learning that you are doing, needs to be carried over to your professional life. This is something that is coming over again, for which when I'm doing hiring, I'm always looking at the project of the people.

Kristina Hoeppner 04:33

Both of you are members of the BMBF project, a Germany federal government funded project, called Artificial Intelligence Supported Observations of EPortfolios (AISOP). What is the aim of the project? How did you think to work with portfolios in that context?

Wolfgang Müller 04:52

I think the starting point was, of course, our experience that we have since now 15 years in our study programme of Media Education and Management, and we love ePortfolios. We think it's a very valuable tool, both for us as teachers, but also for our students. Starting from that we were thinking of applying it on a wider scale in the teacher education programme of our university. This being said is that our teacher education faculty is one of the larger ones here at our university. We're talking about something like 2,000 students that we got here. So it's something like 300 students in a semester. 

Wolfgang Müller 05:27

We love e portfolios, but what we also have to admit is that assessing portfolios is time consuming. I think this is quite natural since we're talking about highly individualised learning processes there, which is something what we love to see, but scaling it to a larger scale than at the size I mentioned before in our teacher education programme is of course difficult. We also had to convince our teachers involved in the teacher education programme that they can manage this kind of assessment, given the kind of pressures, the kind of work they already have to do there. 

Wolfgang Müller 05:58

And this is why we thought about this, such as, 'How can we ease the assessment of portfolios? And how can we also make it easier for teachers to provide formative feedback or a way a time when the portfolio has been created?' So there was the starting point. That's how AI came into the game. And we asked yourselves, 'What roles could AI take over in this kind of an assessment? 

Paul Libbrecht 06:22

Think of it, the simplest thing would be to compare two portfolios, which is happening all the times when you meet maybe two different students or when you think about comparing how do things need to be marked. And this kind of comparison could be well supported, well, I mean, what you can do is read both portfolios up to the end, and then think, 'Okay, well, that thing was not treated, that thing was treated, or this was really intensive, or this was very innovative, or this was very far fetched or very personal.' All these information is something that some artificial intelligence is able to detect, and there should be a possibility to support this using text analysis.

Wolfgang Müller 07:02

I would like to add something there. Because one thing, which is important to mention is, we in our team, we don't believe in fully automatic assessment. So this is not something that we would like to achieve, absolutely not. We see it as our role as teachers to look into the ePortfolios and to find out about the strengths and weaknesses of the ePortfolios and to see and to assess the learning paths as the development of our students by ourselves. But we understand technology as a tool, which may aid us in this kind of an assessment. So taking over for instance, this aspect of identifying average solutions, average kind of mistakes, which we can see there so that we can concentrate on the strength, the individual strengths of specific learning processes, and the results being presented, and also of about some points, where apparently our students need our support, which we can provide in terms of formative feedback then. 

Kristina Hoeppner 07:58

The tool is really there for both sides, the student and the teacher. Can I summarise it as getting an initial view of the portfolio, a little bit written assessment summary, but the student also getting that feedback in regards to their competency or what they should still develop on their project? Is it helping reflection? Is it helping students see where their strengths are, where they might still need some more work or how can it be incorporated into the portfolio? 

Paul Libbrecht 08:28

Correct. These are the goals, but you can formulate this as, for example, would be doing the chat kind of software and say, 'Go and look for this, one single recommendation at a time, but you can also present this in a very different fashion, in a way that, for example, shows that you have covered these topics, but probably you have omitted these topics, or you have been very deep or very innovative on these aspects, on these aspects, this is a strength that you could generalise on some other parts.' These are things that can be provided in a way which is not necessarily one single recommendation at a time using text, but using visualisations. So it's important for us that we present the service, the output of the text analysis in a way that encourages the student into acting, into reflecting, but that does not necessarily explicitly say 'Okay, at to finish you will get a bad mark or this is something that is perfect on this one.'

Kristina Hoeppner 09:27

So what is your data source then or your data set? Do you access a huge data set, say like the ones that we are seeing with Open AI and ChatGPT or is it limited to what your students have said in similar assignments for portfolios in the past so that you have that contextual portfolio view?

Paul Libbrecht 09:47

Multifaceted. The first thing is the topic structures, which we tried to also represent graphically, which has a knowledge representation in itself. The second thing is the course content, and then the portfolios, all of which you can input into some topic recognition system using annotations, using, you know, training. Of course, you would want then to take all the portfolios of any student of the earth, but you would have a privacy problem. 

Paul Libbrecht 10:15

So we try to do things a bit differently. We try to anonymise portfolios and make them into a corpus that is shareable because they're properly anonymised, and we are authorised to make them shareable. We want to provide open content, but there are some limits to this, and for some parts, we will try to use external systems. We will try to use, for example, automatic recognition of images, and these needs to transmit the image to a big system that is one of these large models. 

Paul Libbrecht 10:46

We want to go further and even also try to apply the Open AI large models, but what you find in the world of 'open', in the world of readily available models, transformer models, for example, they allow you to do some matching in a privacy clean way, where you don't have so much worries, where you don't need to send all the pictures to some services, which you cannot control. And we try to make the most possible out of these services.

Kristina Hoeppner 11:15

Which of course is very important in Germany and the larger EU with the GDPR, the data privacy and data sovereignty requirements there. 

Paul Libbrecht 11:24

Correct.

Kristina Hoeppner 11:24

And also the language of course because while we are talking in English here, I assume that a lot of your portfolios are actually in German and maybe also other languages that are used in the same text maybe even. Have you seen any interesting patterns arising from that? How also the responses are to the students and to the teachers in the feedback when you use different languages?

Wolfgang Müller 11:47

We didn't apply our technology in practice as a product yet, we're still in the development phase, first of all, that's one aspect, and also our culture is more that we focus on the German language for the moment. We have single cases there of portfolios and parts of them being presented there in an English language. So this has not been one of our major use cases yet.

Paul Libbrecht 12:07

The German is the dominating thing here and German in itself doesn't have the best models. Being in English would make a lot of things more easy. There is an amount of things which are ready and which are applicable and that's sufficient. We try to make some of our knowledge outputs also multilingual; the concept maps and their terminology, which is [unintelligible] will be multilingual, and definitely English and German, since switching between the two languages is common. So it will not be rare that in our annotation process, we will have to catch phrases which are half English, half in German, because that's normal in computer science. 

Kristina Hoeppner 12:47

Yeah. So you are now about a year and a half into the project. You started it officially at the end of 2021, and Wolfgang, you just mentioned that you are in the development phase and have not gone into production, which I think is scheduled for sometime later this year. Where are you up to now? Are you focusing on the text? Have you already branched out into the visualisations? Are there multiple work streams? Because there are only a few people listed on the project. So I assume it's quite an intense phase for all of your working on it.

Wolfgang Müller 13:20

It is yes. And we also, since we're really targeting to apply our technology in practice, we need some kind of quality of our tools and we have to integrate them into our university systems. One of the major systems there is our Mahara system coupled to our Moodle platform. So one thing we had to achieve there is really to provide interfaces also to access Mahara portfolios, to provide them for analysis. And right now, we got so far that we were successfully able to extract corresponding portfolios with necessary steps based on privacy. So we need for certain steps here in Germany, also the allowance (sic; permission) of students for such kind of analysis, and we implemented our first version there of the analysis pipeline. 

Wolfgang Müller 14:08

So we're doing first experiments here, at the same time, we are designing and experimenting here with dashboard prototypes because we are looking quite intensively there on the requirements of teachers as well as students in terms of what are their needs, what is what they would like to see what kind of feedback they would like to get there?

Paul Libbrecht 14:29

What is it that we can provide, what kind of indicators and we provide?

Wolfgang Müller 14:33

Exactly. First prototypes are on the way of to development and we target to integrate those with our pipeline now over the next months, go for first experimentation in the context of our assessments here at the university.

Kristina Hoeppner 14:47

Fantastic. 

Paul Libbrecht 14:47

So currently, for example, we have the interface working with our development Mahara, but plugging into the public Mahara will not happen before we can convince our privacy gurus that this is okay. And not only the privacy gurus, but also the people who control Mahara, which is in production mode, which is not in development mode. There are an amount of things that need to be there and where we have to build a trust of the students. And the students need just to say, 'Yeah, I'm okay with that privacy going somewhere else. And I want to benefit of these services because of that.'

Kristina Hoeppner 15:17

Wolfgang, you mentioned dashboard prototypes. Can you explain what these are because I'm not quite familiar with that term. I know what the dashboard is, and I know what the prototype is, but kind of meshing them together in your context, that would be great if you gave a brief explanation, please.

Wolfgang Müller 15:32

Our idea is not only to provide informative web pages with single visualisations. Paul was mentioning that before that we are quite thinking visually, in a way. At the same time really investigating the specific kinds of micro tasks in such an analysis. Paul was mentioning one thing, for instance, as a teacher would, of course, like at some point to compare two portfolios, but maybe you would like to compare a portfolio to the average of the other portfolios around. And another task, which you can see there is investigate whether a certain portfolio contains specific aspects, which we think are important, or what we see as important in the learning paths of a student. So these are three different kinds of tasks. And we would like to support all of them appropriately. 

Wolfgang Müller 16:24

That's why we're thinking in terms of dashboards, dashboards being setups of different kinds of visualisations, providing us with an overview, the teacher was an overview first, and allowing you to drill down to the corresponding aspects you would like to investigate in some more detail. This analysis and this design, for this we are using techniques from usability engineering and user experience design. And we are testing this also with the necessary analysis of visualisation tasks, of analytical tasks at the same time. These prototypes are being developed right now. We have first prototypes at hand, and some of them correspond to paper prototype for the moment, and we are evaluating them in user tests with users of our target group.

Kristina Hoeppner 17:11

Fantastic,. I must say, I do love a good visualisation because it often conveys the meaning so much more easily and gives you that quick overview. So I look forward to seeing what comes out of it, and how your teachers and also the students with their own dashboards make use of it, and then take that as a starting point to dive deeper into what has come back as feedback. So the AI is giving some ideas to the students of what they might want to do, what they might want to look into. Can that then also be coupled with feedback from peers or from their instructors, from their teachers? Or do those two things sit separately?

Paul Libbrecht 17:52

They would support the dialogue. We're not sure we want to do this because you can still do it with a screenshot, but being able to share a dashboard visualisation, saying, you know, 'I've reached this level, or I don't know how to go further with this information,' would certainly support the teaching process, would certainly support the peer to peer interactions in an effective way. 

Paul Libbrecht 18:12

So we are under this impression that discussing around visualisation is going to support a teaching process in a very effective way. There are a lot of common points between teachers and students. But it's clear that what the teacher is going to be seeing is at least what the student is going to be seeing. I think that we will try to make it clear that, for example, within a course, within a remoter distance course, there's also a little time to say, 'Okay, we went further. So these are a typical trap that you can make from a visualisation or that you can make from a portfolio. Here, it could be a way out of this trap, for example.

Kristina Hoeppner 18:51

What are the challenges in this artificial intelligence project for you?

Wolfgang Müller 18:57

Being here in Germany, I will start with the legal ones [Kristina laughs]. Not kidding. It's quite quite difficult for us in Germany, trying to respect on the one hand, our legal regulations and also the restrictions which we see here, and applying this kind of technology, this kind of analysis.

Paul Libbrecht 19:15

The mandate of being a public university, certainly comes with another set of legal obligations.

Wolfgang Müller 19:21

And this really means that we have to integrate, we had to think about this aspects of privacy from the very beginning. And to integrate this also in the design of our complete system. This took time, and was also challenging.

Kristina Hoeppner 19:34

On the other hand, I think it is a really good thing to think about the privacy and all these legal aspects so that a model is created that can be used later on, beyond the experimentation phase where one might want to be a bit freer and really make sure yes, this can actually go into production and then doesn't have to have all of that rework needed in order to secure it, in order to make it work with privacy requirements.

Paul Libbrecht 19:59

Right. Other aspects might be the transparency. So the fact that you understand why given topic detection has given you this topic or why you are actually being analysed with a bizarre language or a difficult or a semantically intensive language. All these things require some kind of a understandability of our systems, which we will try to do by, you know, proper documentation and explanations and so on. But which we want also to try to do by openness, by the fact that we offer as much as possible in an open process, an open documented method, an open software, and based on this, we have hoped that the students will be less bothered to contribute their content, and we'll see where, for example, their portfolio will be travelling.

Kristina Hoeppner 20:47

So you've been using portfolios for many, many years, you're now involved in an AI project that uses portfolio data and enhances portfolios. Is there anything that you can't yet fully do with portfolios at this point in time that you'd like to be able to do?

Wolfgang Müller 21:04

I would start with the current projects, what we are targeting there is something like a vision, a dream, which we have, since a couple of years now. Some tools which really assist us on the assessment of portfolios and provide feedback at a time. I should mention that feeling that we always had is that we don't have enough time to really investigative the portfolios in such detail to provide sufficient feedback during the development of the portfolios by the students. We really hope that these tools when we have them at hand that they support us in this specific aspect. Another thing is, of course, we're fans of portfolios, of ePortfolios, and we like to spread the idea. And we like to see it being applied also in other subject fields and by other persons here at our university. And we really hope that our technologies will support this.

Paul Libbrecht 21:54

I would add one thing, which I think has been always a stumbling block with portfolios is that there is this ubiquity of some tools, such as email programs, such as you know, word processing, such as calendar systems. Portfolio should become just as ubiquitous as these ones. You should be able to make a portfolio just by putting a paper into a machine and it will create your portfolio out of that . Is that something like the future, I don't know. But clearly, I would really love to have this portfolio as being something that can be more ubiquitously because it's just as easy as an email.

Kristina Hoeppner 22:30

That would then also support the lifelong learning that students don't see it as this big thing to do, but it becomes part of their daily routine, weekly routine, or monthly routine, and then it's carried on further. So to finish off, we have the quick intro round. So I'd like to ask the three questions of each of you. And we'll start with Wolfgang for the first one. Which words or short phrases, and that is up to three, do you use to describe portfolio work?

Wolfgang Müller 22:59

I see ePortfolios as a really intelligent and effective way for assessment.

Kristina Hoeppner 23:05

A sentence works perfectly as well. Thank you, and, Paul?

Paul Libbrecht 23:10

I'll use a sentence: prepare your career.

Kristina Hoeppner 23:12

What tip do you then have for learning designers or teachers, instructors who create portfolio activities? Paul, do you want to start?

Paul Libbrecht 23:22

I would recommend to include on a steady basis, this reflection aspect, on a daily basis somewhere on the agenda and require this, even if this gets all deleted, require this as part of the process, encourage that the students have to think about the process all along, even if it's just like, 'Oh, it was hard.'

Kristina Hoeppner 23:44

Thank you, and Wolfgang?

Wolfgang Müller 23:46

Think in projects. Think in terms of projects because I think this is where we come from. We like the idea of self directed learning, of project based activities, and are saying 'The best you can do also to explore the strengths, which you can later find and in ePortfolios is think in terms of corresponding activities that you integrate in your learning processes. It doesn't have to be a complete project, but certain elements.'

Kristina Hoeppner 24:11

Now what advice on the other hand, do you have for portfolio authors, for learners, for students?

Paul Libbrecht 24:17

I would say, make it personal. The portfolio is the opportunity for you to transfer the learning to your personal world, and this is an extremely precious proof of learning.

Wolfgang Müller 24:28

I can hardly add something to that. I didn't mention that before that most of our students when we start here with our activities and our study programme, they're scared because they see something not being applied before at school level. They don't know what this is what the expectations are. And we always tell them, 'Guys, this is great.' You can have elements of self direction, you can' as Paul said,' make it personal, and it's a great thing. Try it out. It's great.'

Paul Libbrecht 24:54

I would add one thing, make it personal and make it coherent. Making it personal is the only way to make it coherent, and this way you prove that it's not ChatGPT, which has written it all, from A to Z.

Kristina Hoeppner 25:06

Wolfgang, your students often come scared to the portfolio task, how long does it take them to get over being scared?

Wolfgang Müller 25:14

After the first assessment. I think then they like it, and this is something what we get on a regular basis as feedback that students of higher semesters they tell us, 'Well, we have been afraid, what is this with the ePortfolios, and we hated that a little bit at a time when we had to write it. But now it's so great to have this artefact at hand, very, very often still consulted and look up certain aspects which I investigated there and some details. So it's something which you really accompanies my learning for a much, much longer time.'

Kristina Hoeppner 25:43

That's wonderful to hear about assessment tasks that otherwise just end up in the drawer or are never looked at if they are just a test. 

Kristina Hoeppner 25:50

Thank you so much, Paul, and Wolfgang, for giving us a glimpse into your AI powered portfolio project, and I wish you all the best for the remainder of the project and definitely look forward to seeing what comes out of it. 

Paul Libbrecht 26:06

Thank you.

Wolfgang Müller 26:07

Thank you very much.

Kristina Hoeppner 26:10

Now over to our listeners. What do you want to try in your own portfolio practice? This was 'Create. Share. Engage.' with Professor Dr Paul Libbrecht and Professor Dr Wolfgang Müller. Head to our website podcast.mahara.org where you can find links and the transcript for this episode. 

Kristina Hoeppner 26:29

This podcast is produced by Catalyst IT, and I'm your host, Kristina Hoeppner, Project Lead and Product Manager of the portfolio platform Mahara. Our next episode will air in two weeks. I hope you'll listen again and tell a colleague about it so they can subscribe. Until then, create, share, and engage.